Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment
Authors: Yankai Jiang
Wenhui Lei
Xiaofan Zhang
Shaoting Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Medical Image Segmentation
Vision-Language Pre-Training
Zero-Shot Segmentation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce Malenia, a novel multi-scale lesion-level mask-attribute alignment framework designed for superior zero-shot lesion segmentation in 3D CT scans.

Abstract: Recent advancements in medical vision-language pre-training models have driven significant progress in zero-shot disease recognition. However, transferring image-level knowledge to pixel-level tasks, such as lesion segmentation in 3D CT scans, remains a critical challenge. Due to the complexity and variability of pathological visual characteristics, existing methods struggle to align fine-grained lesion features not encountered during training with disease-related textual representations. In this paper, we present Malenia, a novel multi-scale lesion-level mask-attribute alignment framework, specifically designed for 3D zero-shot lesion segmentation. Malenia improves the compatibility between mask representations and their associated elemental attributes, explicitly linking the visual features of unseen lesions with the extensible knowledge learned from previously seen ones. Furthermore, we design a Cross-Modal Knowledge Injection module to enhance both visual and textual features with mutually beneficial information, effectively guiding the generation of segmentation results. Comprehensive experiments across three datasets and 12 lesion categories validate the superior performance of Malenia.

Published as a conference paper ICLR 2025
UNLEASHING THE POTENTIAL OF VISION-LANGUAGE PRE-TRAINIE FOR 3D ZERO-SHOT LESION SEGMENTA- TION VIA MASK- ALIGNMENT
Yankai Jiang", Wenhui Leilike,
Xiaofan Zhang 1.20,
Shaoting Zhang Ite
'Shanghai AI Laboratory SShanghai Jiao Tong University
{1 (jiangyankai, zhangshaoting @p @ppiiaat jlab. org. en, {wenhul Fernhuiilei,,. xiaofan, zhang) jsjtu. edu.
ABSTRACT
Recent advancements in medical vision-language pre-training models have driven signifi- cant progress in zero-shot disease recognition. However, transferring image-level knowl- edge to pixel-level tasks, such as lesion segmentation in 3D CT scans, remains critical challenge. Due to the complexity and variability of pathological visual characteristics, ex- isting methods struggle to align fine-grained lesion features not encountered during train- ing with disease-related textual representations. In this paper, we present Malenia, novel multi-scale lesion-level mask-attribute alignment framework, specifically designed for 3D zero-shot lesion segmentation. Malenia improves the compatibility between mask repre- sentations and their associated elemental attributes, explicitly linking the visual features of unseen lesions with the extensible knowledge learned from previously seen ones. Further- more, we design Cross-Modal Knowledge Injection module enhance both visual and textual features with mutually beneficial information, effectively guiding the generation of segmentation results. Comprehensive experiments across three datasets and 12 lesion categories validate the superior performance of Malenia.
INTRODUCTION
3D medical image segmentation has witnessed rapid advancements in recent years (Isensee et al. 2021 Tang jet al., 2022 Ye et al., 2023; Liu et al. 2023 Chen et al. 2024 Zhang al. (022)). However, most tate-of- the-art (SOTA) methods are restricted to closed-set setting, where they can only predict categories present in the training dataset and typically fail to generalize to unseen disease categories. Given the diversity and prevalence of new anomalies in clinical scenarios, along with the challenges of medical data collection, there is an increasing demand for zero-shot models capable of handling unseen diseases in an open-set setting.
The advent of vision-language pre-training methods, particularly CLIP (Radford et al.) 2021), has illumi- nated a new paradigm for remarkable zero-shot object recognition. This breakthrough also paves the way for significant advancements in zero-shot disease detection and diagnosis. Numerous recent methods (Huang) lt 2021 Tiu et al. 2022 Wu et al. 20233 Phan et al.) 2024 Hamamci et al. 2024) align visual and textual Teatures of paired medical image-report data, enabling transferable cross-modal representations. However, leveraging the zero-shot capability of lision-language pre-training for 3D lesion/tumor segmentation re- mains a scarcely explored area. This extension is nontrivial and faces two obvious challenges: (i) The substantial gap between the upstream contrastive pre-training task and the downstream per-pixel dense pre- diction task. The former focuses on aligning image-level global representations with text embeddings, while
"Work under support from the Shanghai Artificial Intelligence Laboratory. Corresponding authors. Codes are available at allatps::/gittiet http://tttuu..o//aaaeee.e wttp://w.oo.a..o//ooooaaaa

---

Published as a conference paper at ICLR 2025
the latter requires fine-grained lesion-level visual understanding. This inherent gap necessitates the develop- ment of more advanced fine-grained vision- langual alignment techniques that can facilitate the perception of nuanced, patient-specifie pathological visual clues based on the text descriptions. (ii) Lesions can exhibit significant variations in shape and size, and may present with blurred boundaries. Models often struggle when encountering unseen lesion types due to their out-of-distributoon visual characteristics. Simply using text inputs, such as raw reports (Boecking et al. 2022 Tiu et al. [022]] Hamamci et al. (002). or relying on common knowledge of disease definitions (Wu et al. 2023| Jiang et al. (2024), is insufficient for learning generalized representations needed to segment novel lesions not mentioned in the training dataset.
Motivated by the aforementioned limitations, we introduce Malenia, a novel multi-scale lesion-level mask- attribute alignment framework for superior zero-shot lesion segmentation. Malenia first leverages multi- scale mask representations with inherent boundary information to capture diverse lesion regions, then matches fine-grained visual features of lesions with text embeddings, effectively bridging the gap between the contrastive pre-training task and the per-pixel dense prediction task. To learn extensible representations that are robust to the outoo--distriuutinn visual characteristics of unseen lesions. we incorporate domain knowledge from human experts to structure textual reports into descriptions of various elemental disease visual attributes (e.g., shape, intensity, location). Despite the significant variability among lesions, these fundamental attributes are shared across different diseases and are often represented similarly in images. By aligning mask representations of lesions with their corresponding visual attributes, the model mimics the decision-making process of human experts, explicitly linking the visual features of unseen diseases to the intrinsic attributes learned from seen lesions. Furthermore, we propose a novel Cross-Modal Knowledge Injection (CMKI) module in Malenia, inspired by the observation that visual and textual embeddings, after feature alignment, are complementary and can mutually reinforce each other. The CMKI module updates both mask and attribute embeddings to facilitate fine-graine multi-modal feature fusion. We leverage both enhanced mask and attribute embeddings to generate predictions by matching query features with image fea- tures, and then ensemble these predictions to produce the final segmentation results, demonstrating improved performance for both seen and unseen lesions. To thoroughly validate the effectiveness of Malenia, we eval- uate its segmentation performance on both seen and unseen lesions using the MSD (Antonelli al., 2022). KiTS23 (Heller et al. 2023). and a curated real-world in- to-huuse dataset. Malenia consistently outperforms state-of-the -he- art methods across 12 lesion categories. Our contributions can be summarized follows:
We present Malenia, a novel multi-scale lesion-level mask-attribute alignment framework that cap- tures extensible multi-modal representations for significantly improved zero-shot lesion segmenta- tion by effectively matching the fine- tme-grained visual appearances of new diseases with the textual representations of various fundamental pathological attributes.
Malenia introduces a novel Cross-Modal Knowledge Injection (CMKI) module that enriches both mask and text embeddings with mutually beneficial information through feature fusion, leveraging their complementary strengths to further enhance lesion segmentation performance.
State-of-the-ar lesion segmentation performance in the zero-shot setting across three datasets. Malenia significantly outperforms previous methods, and key ablation experiments demonstrate the effectiveness of our strategies in handling lesions with varying characteristics.
2 RELATED WORKS
Medical Lingua-Languuge Pre-training. Numerous methods (Huang et al. 2021 Tiu et al., 2022 Wu et al., 2023 Phan et al., [2024] Hamamci et al.) 2024 Lai et al., (202)) build upon CLIP and align medical images with their corresponding reports or disease definitions to enable zero-shot discase classification. However, these works focus on single body parts (e.g., chest), which limits their applicability in broader medical contexts. Additionally, these methods primarily adhere to a paradigm that matches text embeddings with global image-level or patch-level semantics, originally designed for classification tasks. When adapting their
2

---

Published as a conference paper at ICLR 2025
zero-shot capability to fine-grained segmentation tasks, mismatches can occur due the model's inability to align text embeddings with detailed pixel features. A recent work, CT-GLIP (Lin et al. (024) uses organ- level vision-language alignment for zero-shot organ classification and abnormality detection, but due to architectural limitations, it requires fine-tuning tuning with a segmentation head and lacks zero-shot segmentation capabilittes. In contrast, we extend the zero-shot capability of vision- -aiigage pre-training from image-level to pixel-level by aligning lesion features with fundamental multi-aspect disease visual attributes.
Zero-Shot 3D Medical Image Segmentation. Motivated by the impressive zero-shot performance of SAM (Kirillov al. 2023) and SAM 2 (Ravi et al. 2024 in natural images, numerous studies have eval-
uated their application in medical image segmentation (Wald et al. 2023 Zhang et al. 2023) Huang et al.)
2024b Yamagishi et al., 2024) and explored effective adaptations of SAM or SAM 2 medical datasets (Ma
et al. 2024 Guo et al. 2024] Shen et al., 2024] (Zuu et al.) 2024, Shaharabany & Wolf (024). Nevertheless,
these SAM-based medical image segmentation methods require prompts (points, bounding boxes, masks)
sampled from ground truth during testing, which demands significant expertise and is often impractical in
real-world clinical scenarios. As a result, prompt-free SAM adaptation methods (Hu et al., 2023 Zhang
Liu 2023 Cheng et al.) 2024: Aleem et al. (2024) have also been proposed. To the best of our knowledge, although the current SAM-based 3D medical image segmentation methods demonstrate promising zero-shot performance in segmenting certain organs in CT scans, they have not been evaluated or proven effective when confronted with unseen lesions that have less defined structures. Apart from SAM- models, a self-prompted method, ZePT (Jiang et al.] (024)), achieves competitive zero-shot tumor segmentation per- formance by matching class-agnostic mask proposals with text descriptions of general medical knowledge. However, it overlooks patient-specific information from reports, leading to compromised zero-shot perfor- mance. Our approach goes a step further, achieving significantly superior zero-shot lesion segmentation performance without the need for complex visual prompts.
Laniuage-Guuided Medical Image Segmentation. Recently, several methods also incorporate vision and language information to enhance medical image segmentation (Li et al. 2023 Liu al., (023) Huang et al., 2024a). Notably. LViT (Li et al., 2023) leverages medical texts to compensate for quality deficiencies in image data and guide the generation of pseudo labels in semi-supervised settings. RecLMIS (Huang al., 2024a) introduces conditioned reconstruction to explicitly capture cross-modal interactions between medi- cal images and text, leading to enhanced performance. However, these methods primarily focus on fully- supervised and semi-supervised settings, aiming to enhance segmentation performance for seen categories by combining visual and textual features. In contrast, Malenia targets the zero-shot setting, aiming to de- velop generalized and robust textual features, such as disease attributes, along with novel vision-language alignment strategy to enhance segmentation performance for unseen lesions.
3 METHOD
Fig. illustrates the pipeline of Malenia. It is built upon the recent mask-based segmentation backbone Masmerrrmer (Cheng et al. 2022). Below, introduce (i) our novel multi-scale mask-attribute alignment strategy (8ec(.1] that effectively aligns lesion representations with text embeddings of fundamental disease attributes; (ii) the Cross-Modal Knowledge Injection module (Sec 3.2) for multi-modal feature representa- tion enhancement; and (iii) the overall training objectives and inference details (Sec. 3.3). We provide the preliminaries, discussing Farmer and the problem formulation of zero-shot lesion segmentation, as prior information required for our work in Appendix A
3.1 MULTI-SCALE MASK-TEN CONTRASTIVE LEARNING
Given a set of B image-report pairs, ID = ((I,, R1), (IB, RB)), we aim to establish lesion-level align- ment between the mask and text representations within the multi-modal feature space. We first adopt a 3D
3

---

Published as a conference paper ICLR 2025
Muln- Contratiiv fate.
Structurel Regoots
Embesddnng
Malched Mask and A Atttluuto Tokens
Loiis Liver Tamee
Locatie:: Liker
Tixt Eocoder
Multi- Scale Fine-Graine Grained Mask--A ttribute A lignment
Two-teunt
Shape Bausd-lker
Ensemble
Dendtty Hepredence
Kask.
Donitt Yartutiont...
Fahurorntent Statont
Serfare Charateriatio
NMMakk Volerns
Block
Block
Block
Relulienaip Organnc
Specill Fedtane:
Cross- Modal Knowledge Injection
[Rg
SCITA
K
UP Mlask
Up Mask
Uie Mark
30 Inpot image
Aator
Lyiwe
Kuice
k.
Irrerr
Image' Encoder
High Resalutin Image Fechere
Image Decoder
4 Splenvi
Figure 1: Overview of Malenia. The key contributions of our work are two simple but effective designs: the Mask-Atrribute Alignment and the CMKI module.
Multi-scale
image encoder to extract high-level features f from the input 3D images I. Then a 3D image decoder grad- ually upsamples f to generate multi-scale high-resolution per-pixel image features F E TR. H xW W xD' D'x Here, H' W D' and C denote the height, width, depth and channel dimension of F1, respectively, which vary depending
the resolution level. Subsequently, mn). along with successive feature maps 1" from the initial three layers of the 3D image decoder into a Transformer decoder with 3 blocks in round-robin fashion process the mask tokens. At each block i E [1. 3. the mask tokens M undergoes a series of faser-wise attention refinements, including attention and self-attention as follows:
feed N learnable mask tokens (queries)
.M = {m1,.
M(i) cross CrossAAtm(g (0) (+)) M = SelfAttn (+) cross)
(1)
Here, k((+) and (x)) denote the query matrices of mask tokens and the key and value matrices of image features from the 3D image decoder at the i-th resolution level, respectively. During the cross-attention phase, mask tokens interact with image features, focusing on the specific regional context within the image. In the self-attention phase, mask tokens interact with each other to enhance the understanding of relationships between different anatomical areas. Building on the segment-level embeddings of mask tokens, we establish fine-grained alignment between mask and text representations through a contrastive learning approach. This vision-language alignment Malenia has three novel key components:
(1) Utilization of Multi-Scale Features. Existing methods (Jiang et al. 2024 Lin et al. 2024) over- look the advantage of leveraging multi-scale visual features during cross modal alignment. In contrast, we match the hicrarchical mask token embeddings from different Transformer decoder blocks with text fea- tures. Specifically, the mask tokens interact with image features whose dimensions are set as (h² w², D³) (H/32, W/32, D/32), (H/16, W/16, D D/16). (H/8, W/8, D/8) for blcoks i = 1,2, 3, respectively. This variation feature resolution across blocks ensures mask-text alignment at different scales, which is crucial for segmenting classes with large size variations, such as tumors.
(2) Dissecting Reports into Descriptions Fundamental Disease Attributes. Human experts make di- agnoses by carefully analyzing key image features that describe distinctive lesion attributes (e.g.. shape and density) across different disease classes (Ganeshan et al.) 2018 Nobel et al. (2022). Drawing from this in- spiration, we consult medical experts and decompose reports into structured descriptions of 8 shared visual attributes of disease imaging characteristics, including location, shape, density, density variations, sur- face characteristics, enhancement status, relationship with surrounding organs and specific features.
4

---

Published as a conference paper at ICLR 2025
Specifically, we adopt semi-automatic pipeline to transform patients' reports into structured descriptions. First, we prompt the Large Language Model (LLM) GPT-4 (Achiam et al. 2023) to extract descriptions re- lated to the lesions from the findings section of each report. Then, two experienced radiologists collaborate to review, correct, supplement, and expand the GPPTgennraaee visual descriptions based on the CT images and corresponding original reports, Leveraging disease attribute descriptions offers two key advantages. First, disease attributes provide fine-grained prior knowledge about the visual manifestations of patholo- gies, inherently improving alignment with target diseases. Second, new diseases can be described using the elemental visual attributes of previously seen discases, thereby enhancing the model's s zero-shot capability.
Contrastive Loss. Given the multi-scale lesion-level mask embeddings M and visual attribute descriptions, we construct multiple positive and negative pairs, which are then used to learn to op- timize a distance metric that brings the positive pairs closer while pushing the negative pairs away. At the i-th resolution scale, we obtain N binary mask proposals BM E [0, 1 x xW'xD'xN X by a multplicationn operation between the mask embeddings and image features followed by Sigmoid. We then apply bipar- tite matching between the upsampled binary mask proposals and the ground truth masks, following (Cheng ltt al. 2022), to select S foreground mask tokens that correspond to the S lesions in the input 3D CT image in a one-to-one manner. Next, we feed the ground truth descriptions of all visual attributes into the text encoder followed by a MLP layer to acquire the text embeddings T {tt..... .t....... .ta), where R denotes the number of different attribute descriptions. In our pipeline, ground mask tokens that do not have a matched ground truth segment. Therefore, we add an extra learnable embedding, lo RCC representing "no lesion found". Finally, each foreground mask token is paired with its corresponding eight distinct text features, forming multiple positive sample pairs. While each background mask token is paired with to. For the j-th mask token, let P2 [k](m, t&)) positive pair) represent the set of all its positive text embedding indices, and let M, = [m 4x) is a negative pair} represent the set of all its negative text embedding indices. We calculate the pairwise similarity score S(m,. t&) between the j-th mask token my and the k-th text embedding ty as dot product, normalized by temperature parameter 7, given by S(mj, (x)
(3)
also require background category for the N - S back-
42 The lesion-level mask-attribute alignment at the i-th resolution level is
refined using a contrastive loss function ((8) designed to maximize the similarity scores of positive pairs while minimizing those of negative pairs. Since each lesion has eight positive text embeddings of visual attributes, our framework includes multiple positive pairs for each foreground mask token. The commonly used N -pir loss (Sohn 2016) and InfoNCE loss (Oord et al. 2018), which handle a single positive pair and multiple negative pairs, not suitable. Therefore, adopt the Multi-Positive NCE (MP-NCE) loss (Lee et al. 2022 to define the c(8) as:
exp (5(m)) (1) .tx)
Σ EsEP, log exp (S(m))) (4) +x)) + Enen, exp (S(m)) tn))
(2)
N
(3)
In this way, we explicitly brings lesion-level mask embeddings closer to their corresponding attribute fea- tures while distancing them from unrelated ones. This enables the textual features of each attribute a bridge between the visual features of different diseases, effectively improving the model's 's zero-shot per- formance by linking the attributes of unseen lesions with base visual knowledge. Moreover, we apply C(ú aim to output mask tokens from each Transformer decoder block, utilizing multi-scale feature maps. The multi- scale lesion-level mask-attribute alignment loss is formulated as Lyim - E/m c(i) where L 3 denotes the number of Transformer Decoder blocks.
as
3.2 CROSS- KNOWLEDGE INJECTION MODULE
In Malenia, we introduce a novel Cross-Modal Knowledge Injection (CMKI) module to generate the final segmentation predictions. This module enriches the features of one modality by incorporating information
5

---

Published as conference paper ICLR 2025
Slegel
Miskk Lokens
Stop
Sleg-Il
Slep TV
MMK modile
Manching by Smilany Soore
Mask Prodictions
Know wage
Stored Atmbute Embeidiiggs
Disease
Disease Allributes
Mark Tokens 64 M-
Lacaiza
Dasily Postame Reckyeedd
Xizza Tunar
K kidmy Cyst
Imaae Encoder
3D Image Decoder
Rackground
High Resolution Feslunes
Figure? 2: Overview of the inference process of Malenia. (1) Step-I: Image Partitioning via Mask Tokens. Test CT images are divided into regions, each represented by mask tokens. (2) Step-Il: Mask-attribute matching. Each mask token associated with stored attribute embeddings, (3) Step-III: Cross-modal fusion and mask prediction. Information from mask tokens and text embeddings is fused to generate segmentation masks. (4) Step-IV: Disease identification via attribut-querrying The Clinical Knowledge Table links the predicted attributes to specific disease categories for precise diagnosis.
from another, enabling deeper understanding and improved feature representations. Specifically, we fuse the output mask tokens m,, E [V] from the last Transformer decoder block with their corresponding positive attribute embeddings. As shown in Fig. (a), for each mask token, we first concatenate all its corresponding attribute embeddings j j E IP;, and transform them into single text embedding using an MLP layer to obtain the textual features for comprehensive description of the mask token m,. Then accomplish deep fusion between two modalities through a series of cross-attention and self-attention layers:
rh: SelfAttn (CrossAttn(gm: hit, 02) - -ms), i = SelfAttn (CrosssCCCsssss km vm) + 1:)
(3)
Here, 9m km and 2m represent the query, key, and value matrices of the mask tokens, while kt, and et represent those of the attribute embeddings. The deep fusion of vision and language offers two key benefits: 1) Mask representations are enriched with textual information from language models, resulting in more context-aware segmentation. 2) Text embeddings enhance their descriptive capabilities by attending to visual features, enabling segmentation conditioned on specific text prompts.
We leverage both enhanced mask tokens and text embeddings to generate predictions. By capitalizing on fine-grained vision-language alignment, formulate semantic segmentation tween representative multi-modal query embeddings and pixel-level image features. Given the N enhanced mask tokens m e NxC and text embeddings E NxC, as well as the output pixel-level image features Fl E H W DxC from the last 3D) image decoder layer, we apply linear projections & to generate Q (query) and K (key) embeddings
matching problem be-
Qm = (m) E NXC, oz NXCC, K= K=
E "XWWDDD
(4)
Then, the mask predictions could be calculated by the scaled dot product attention:
QmKTT @HHWWDDNN VC
QKKT VC
Mask
Mask,
BRWWWDXN
(5)
Here, Maskm and Mask, refer to the two-branch output mask predictions derived from mask tokens and text embeddings, respectively. VC is the dimension of the keys as a scaling factor. We ensemble these mask predictions Mask M (L Maskm. Maska).). Here 31 and fg are weighting factors for the vision and language branches, respectively, set to 0.5 by default. The final segmentation results are obtained by applying the Argmax operation along the channel dimension of the Mask.
6

---

Published as a conference paper at ICLR 2025
3.3 TRAINING OBJECTIVES AND INFERENCE
Overall Loss Function: We adopt a composite loss function that balances mask-attribute alignment and mask segmentation. Specifically, we use the dice loss Cdice to supervise the segmentation pre- dictions of mask tokens matched with ground truth at each feature level i, achieving deep supervision: Ldeep - Σ C(i) diice. Additionally, we have the similarity loss aism defined by Eq. (2) for aligning mask tokens with attribute embeddings. For the final segmentation results generated by both visual and
textual features, we utilize the binary cross-entropy loss and dice loss: Leeg = a1 Lce + agLddie.. We set
a1 = 2.0 and a22 = 2.0. The overall loss function is weighted sum of these components:
L Ldeep + X2 + Xzhseg-
(6)
where A1 A2 and 0a are weighting factors balancing the contribution of each loss component to the overall training objective. We set À A2 = A3- 1.0 as default.
Inference: Fig. 2 illusstrates the model testing flow. To enable convenient and flexible inference, we store the text features of all visual attribute descriptions after training, climinating the need for the text encoder during testing. The inference process can be divided into four steps. Step-I: Mask tokens attend to different regions among the images, partitioning the image into different regions. In this process, different lesion regions in the image are captured by specific mask tokens. Step-Il: We compute the similarity between the N mask tokens and the stored attribute embeddings, after which the mask tokens are matched to the text embeddings based on these similarity scores. Step-III: The matched mask tokens and text tokens are fed into the CMKT module for further feature fusion. Then, the information from mask tokens and text embeddings are combined to produce the final mask prediction. Step-IV: The text attributes of cach mask are used to identify the specific category. This is achieved by querying a Clinical Knowledge Table that records the relationships between each disease and its attributes. The Clinical Knowledge Table are given in Table 11
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets. We utilize annotated datasets encompassing lesions from 12 diseases and 6 organs, sourced from both public and private datasets. (1) Public datasets: We consider the MSD dataset (Antonelli et al. 2022)
and KiTS23 (Heller et al. 2023) dataset. Two senior radiologists supplemented the annotated lesions MSD and KiTS23 datasets with diagnostic reports and corresponding lesion descriptions of the eight visual attribute aspects. We also collect private d that includes four lesion types: liver cyst, gallbladder tumor, gallstone, and kidney stone. For Iraining, we use the following datasets: 1) colon tumor, lung tumor, liver tumor, and pancreas tumor from the MSD dataset; 2) kidney cyst from the KiTS23 dataset; 3) gallstone from our private dataset. For shot testing, we adopt: 1) hepatic vessel tumor and pancreas cyst from the MSD dataset; 2) kidney tumor from the KiTS23 dataset; and 3) liver cyst, gallbladder tumor and kidney stone from our private dataset. Details of all datasets, the annotation process for disease attribute descriptions, and preprocessing are provided in Appendix B
Evaluation Metrics. We adopt standard segmentation metrics, including the Dice Similarity Coefficient (DSC) and Normalized Surface Distance (NSD). Additionally, we report the the computational efficiency evaluation, which are detailed in Appendix G
Implementation Details. We use nnUNet (Isensee et al., 2021 as the backbone for 3D image encoder and 3D image decoder. We choose Clinical-Boc -errttrrrrrrrte (Alsentzer al. 2019) the text encoder. We employ Adam
"This retrospective study was approved by the ethics committee of the First Hospital of China Medical University; the informed consent requirement waived. All diagnostic reports in the private datasct used in this study have been fully anonymized, with all caleent-ieentifialee information and private details removed.
7

---

Published as a conference paper ICLR 2025
Table 1: Segmentation performance (%) of unseen lesions on the MSD, KiTS23, and our in-house dataset. it denotes methods that adopt SAM SAM2 for zero-shot medical image segmentation, and indicates methods implemented using the official code. All competing methods are trained on the same dataset. The best performance is highlighted in light blue.
MSD
K7TS23
In-house Dataset
Method
Hepatic Vessel Tumor
Pancreas Cyst
Kidney Tumor
Liver Cyst
Kidney Stone
Gallbladder Tumor
DSC1
NSD1
DSCt
NSDf
DSC1
NSD7
DSCT
NSD1
DSCT
NSD
DSC*
NSD
SAM; (hahaarabary Wolf 2024
35.76
45.83
37.17
49.26
35.45
41.33
34,99
40.88
24.14
31,92
28.08
36,38
SAM2 Yamagiihi al 2024
35.93
45.88
38.42
50.85
35.67
41.88
35,29
41.25
25.50
33.74
28.57
36,62
SaLIP+ Alleent et ai 2029
39.65
48.71
41.92
53.06
38.64
44.91
37.71
44,26
27.24
36.61
30.84
38.97
H-SAM Cenng
2024
45.58
54.24
46.87
57.91
44.21
50.39
43.75
50.20
29.23
38.11
32.17
40.05
ZePT= Diang et at 21R4)
53.12
63.25
53.35
63.50
46.82
52.44
51,64
57.36
33.97
42.42
35.48
43.23
Malenia
59.52
69.60
60.91
70.28
54.96
60.60
61,85
70.93
43.05
52.95
47.35
55,79
optimizer (Loshchilov & Hutter 2017) with a warm-up cosine scheduler of 40 epochs. The batch size is set to 2 per GPU. Each input volume is cropped into patches with a size of 96 x 96 x 96. The training process uses an initial learning rate of le-A, momentum of 0.9, and weight decay of le-5, running on 8 NVIDIA A100 GPUs with DDP for 4000 epochs. The number of the mask tokens N is set as 16.
4.2 MAIN RESULTS
Zero-shot segmentation performance on unseen lesions. Table presents the zero-shot lesion segmen- tation results on unseen datasets from various institutions, encompassing wide range of lesion types. All available CT volumes in these tasks are directly used for testing. We compare Malenia with five state-of-
the-art zero-shot medical image segmentation methods: the self-prompted ZePT (Jiang et al. 2024). the prompt-free SAM-based method H-SAM (Cheng al. 2024), the fine-tuning-free method combining CLIP and SAM, SaLIP (Aleem et al. 2024), and two methods (Shaharabany & Wolf [2024] Yamagishi et al. 2024) that respectively adopt SAM and SAM 2. Malenia achieves superior performance across all the datasets, substantially outperforming the other five methods. The weak performance of SAM (Shaharabany & Wolf, 2024) and SAM 2 (Yamagishi et al. (024) can be attributed to the inherent variability of segmentation tasks in different clinical scenarios. Without target data for fine-tuning and accurate manual prompts, the SAM and SAM 2 struggle with lesion segmentation in the zero-shot setting. Additionally, creating accu- rate prompts requires domain knowledge from medical experts, which is often limited and time-consuming. SaLIP leverages the capabilities of CLIP to refine automatically generated prompts for SAM. However, its performance is still constrained by CLIP, which focuses on aligning image-level global features while ne- glecting fine-grained local lesion semantics, resulting in suboptimal performance for lesion segmentation. H-SAM achieves better results by incorporating enhanced mask attention mechanisms in two-stage de- coding process. However, it does not explore cross modal feature representations. Without the rich textual knowledge from language models, its zero-shot segmentation performance is significantly hindered. ZePT is trained using both organ and lesion labels to generate anomaly score maps as prompts, and it aligns mask features with common medical knowledge to enhance zero-shot tumor segmentation. However, overlooks the patient-specffic multi-aspect elemental attributes shared across different diseases, which limits the scala- bility and generalization of its learned representations. In contrast, Malenia learns fine- fine-grained lesion-level mask-attribute alignment to link unseen lesions with base knowledge and leverages both visual and textual context for advanced cross cross-mode modal understanding, resulting in at least a 6.40% improvement in DSC on MSD, a 8.14% improvement in DSC KiTS23, and 9.08% improvement in DSC on the in-house dataset.
Segmentation performance on seen lesions. We also evaluate the segmentation performance of Malenia on seen lesions in the fully-supervised setting. We compare Malenia with SOTA medical image segmentation
methods, including UNen (Chen et al. (021), nnUNet (Isensee et al. 2021). Swin UNETR (Tagg et al.)
2022). and the Universal Model (Liu al.
2023
As shown in Table Malenia outperforms competing
8

---

Published as a conference paper ICLR 2025
Table 2: Segmentation performance (%) of seen lesions on the MSD and KiTS23 dataset. We compare Malenia with SOTA supervised methods and report the 5-fold cross-validation results. means implemented from the official code and trained on the same dataset. The best performance is highlighted in light blue.
MSD
KrTS23
Method
Colon Turnor
Panereas Tumar
Liver Tumor
Lung Tumne
Kidney Cyst
DSCT
NSD1
DSC1
NSDJ
DSC1
NSD1
DSC1
NSD1
DSC1
NSD1
TransUNete
44.78=16.21
54.14+15.67
58 85+10.25
54.72+11.59 59 60.055529 29
72,88+5 98
67. 37.13+6.08
68 08.8947.22
4844444.04
52.32+15.62
unLNeet
47.02=15.85
57.36=14.33
37.97+10.54
61.3345.01
73.27+5.44
89.50+5.66
71.39s6.55
48.76+13.82
52.96#15.19
Swin UNETR+
46.87216.12
55.28a15.52
38,72+10.33
54.01=11.67
62.3724.88
74.7545.00 09
68 98.5555667
67
710046682
48.06214.26
52.11416.05
Universal Modale
51.02=14.62
60.93:13.36
42.40=9.54
18.54
64.2523.94
77.06x4.21
67 67.27+5.71
69.33:6695
50.25+12.224
54.17=13.53
Malenia
53.55=13.49
62.41=12.81
43.30:9.29
59.63:10.55
65.18:3.74
78.95=4.03
70.9625.56
72.34:6.29
5166:11884
55.1=122999
Tmage
Image
Uspersal NOOES
Figure 3: Qualitative visualizations of Malenia and other competing methods on both unseen and seen lesions. The segmentation results, presented from top to bottom and left to right, include Hepatic Vessel Tumor, Pancreas Cyst, Colon Tumor, and Lung Tumor.
baselines on seen lesions, achieving an average improvement of 1.46% in DSC for tumors on the MSD dataset (Antonelli et al. 2022) and 1.35% in DSC for kidney cysts on KiTS23 (Heller et al. 2023). Most segmentation baselines (e.g-, TNet, nnUNet, and Swin UNETR) focus solely visual features and overlook the semantic relationships between different lesions. Although the Universal Model incorporates text embeddings of category names to learn correlations between anatomical regions, it lacks tineggrieed information from both the vision and language domains. In contrast, Malenia outperforms these methods by aligning fine-grained lesion-level semantics with comprehensive, patient-specific textual features and enhancing mask representations through cross-modal knowledge injection. This improvement demonstrates that our novel strategies also enhance the segmentation of seen lesion categories.
Qualitative Comparison. As shown in Fig. 3. Malenia accurately segments various types of lesions across diverse organs, having substantially better performance in segmenting both seen and unseen lesions than the other methods. Most competing methods suffer from incomplete segmentation and false positives. In contrast, Malenia produces results that are more consistent with the ground truth. This further demonstrates the superior lesion segmentation capability of Malenia on datasets with highly diverse lesion semantics.
4.3 ABLATION STUDIES
Multi-scale mask-attribute alignment strategy ablation. We validate the effectiveness of different com- ponents of the multi-scale mask-attribute alignment strategy both seen and unseen lesion datasets, as detailed Table 'Baseline' refers to the naive single-scale mask-report alignment performed at the last Transformer decoder block. We gradually enhance the bascline by (S1) enriching raw reports with struc- tured eight visual attribute descriptions of the lesions; (S2) utilizing multi-scale features for cross modsl muuli--postiiv contrastive learning using multiple
alignment; (Ss) formulating mask-attribute alignment mask-attribute positive pairs, rather than a single mask-report positive pair. Each component contributes the remarkable segmentation performance of Malenia. Enriching raw reports with visual attributes of
9

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study of the multi-scale mask- attribute alignment strategy.
Table 4: Ablation study of the Cross-Modal Knowledge Injection module.
Module
Phrsncas Tumne
MSD
K5TS23
Kalsey Tunee
Cuilhialke Tumie
Module
Pansrcas Tumis
KrTS21 (ussen))
Kilsey Tarner
In-latoc:: Tistact (ensorn)
Gallitaskier Tunte
196.
NST1
NS11
[156.1
NX11
TE MT DF
DSC1
NSP1
DSC
NSD1
[ISC]]
NSD1
Baseline
St.7e100300
54. 24e3l11100 63
47:05
53.47
43.81
22
54
147991113 52
47.34
36.33
44.02
59 59.83816.13
35 13.661111
1:47
48.69
54.32
16.35
46.20
40.274:001
S6. 16.34211.25
49.00
55.14
39.65
47.42
40.4689.95 55
56.52a:1111
1.15
49.63
53.99
39.74
43.79
41.22z99.0 .22
17.15211.10
52.55
58.50
43.07
$1.23
+-i 5t
41 18.864988 80
97.
79--1
1 09
5188
57.58
42.51
50.67
40.5029995
56.6211.14
51.77
57.83
41.28
49.39
1.5 5)
42.2149:71
58.244:0.96
53.54
39.46
42.00
$2.91
42.46-948
58. 96-96-1000 10.70
$2.87
58.07
45.00
51.05
+ S5- S.
43.3049.29
30.8
54.96
08.40
33.99
43.30:9.29
50.33:10.88
$4.96
68.68
47.35
58.99
lesions (S1) helps the model leverage pre-established knowledge of the diseases' visual manifestations to enhance the alignment of fine-grained image features with the representations of target diseases, thereby im- proving segmentation performance. Multi-scale cross-modal alignment (S2) leverages multi-level features accurately capture and segment both seen and unseen lesions across various sizes, which is essential for handling the shape and size variations of lesions. Furthermore, instead of combining the eight visual attribute descriptions of each lesion into a single paragraph and then extracting the text features (S1), we extract the text features for each visual attribute description separately (Sg). As a result, the mask embeddings for each lesion is paired with eight distinet text features, forming multiple positive sample pairs. Simply treating a lesion's 's attribute descriptions as a single paragraph yields only one positive sample pair for each foreground mask token. Consequently, reports of other lesions that share some of the same attribute descriptions are treated as negative samples, leading to compromised feature representations. In contrast, our formulation of the multi-positive contrastive learning (Sg) focuses on establishing comprehensive and extensible correla- tions between pathological features and fundamental disease attributes. This enables the model to associate the visual cues of unseen lesions with foundational visual knowledge, allowing for effective segmentation of new diseases by translating their complex visual features into elemental attributes shared with seen diseases. This significantly enhances zero-shot lesion segmentation performance.
Ablation of the Cross-Modal Knowledge Injection module. We examine key components in the proposed CMKI module in detail, including (1) the significance of deep fusion (DF), which is designed to enhance the representation of mask tokens and text embeddings; (2) the effectiveness of leveraging both text embed- dings (TE) and mask tokens (MT) to generate predictions. The results are shown in Table 4 Comparing the results of the first three rows with the last three rows (highlighted in light red), it is evident that deep fusion significantly improves performance, whether using only text embeddings, only mask tokens, or both for segmentation result prediction. This observation shows the importance of enabling cross-modal informa- tion interaction. Furthermore, whether or not deep fusion is applied, using both text embeddings and mask tokens for segmentation prediction, combined with ensembling the results, consistently outperforms using only unimodal token embeddings for mask prediction. This confirms the superiority of leveraging the com- plementary strengths of both visual and textual embeddings to further enhance segmentation performance.
5 CONCLUSION
In this work, we propose Malenia, a novel vision-language pre-training method designed for 3D zero-shot lesion segmentation, which incorporates an innovative multi-scal, lesion-level vision-language alignment strategy. Inspired by the image interpretation process of human experts, we transform patient-specific re- ports into structured descriptions of disease visual attributes and then match them with mask representations. Additionally, we introduce a novel Cross- Knowledge Injection module, which enhances cross-mode representations and leverages the complementary strengths of both visual and textual features to further improve segmentation performance. Extensive experiments demonstrate that Malenia consistently outper- forms previous state-of-the-an approaches across diverse datasets for 3D zero-shot lesion segmentation. We hope this work inspires further innovation in this challenging yet promising research area.
10

---

Published as a conference paper ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
Sidra Aleem, Fangyijie Wang. Mayug Maniparambil, Eric Arazo, Julia Dietlmeier, Kathleen Curran, Noel EO' Connor, and Suzanne Little. Test-time adaptation with salip: A cascade of sam and clip for sero-shot medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 51 5184-5193, 2024.
Emily Alsentzer, John Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03322 2019.
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoemn Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature communications, 13(1):4128, 2022.
Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- et al. Making the most of text semantics to improve biomedical vision- language processing. In European conference on computer vision, pp. 1-21. Springer, 2022.
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang. Le Lu, Alan L Yuille, and Yuyin Zhou Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306, 2021.
Jieneng Chen, Yingda Xia, Jiawen Yao, Ke Yan, Jianpeng Zhang, Le Lu, Fakai Wang, Bo Zhou, Mingyan Qiu, Qihang Yu, et al. Cancerunit: Towards a single unified model for effective detection, segmentation, and diagnosis of eight major cancers using a large collection of et scans. In Proceedings of the IEEE/CVF International Conference
Computer Vision pp. 21327-21338 2023.
Jieneng Chen, Jieru Mei, Xianhang Li, Yongyi Lu, Qihang Yu, Qingyue Wei, Xiangde Luo, Yutong Xie, Ehsan Adeli, Yan Wang, et al. Transunet: Rethinking the u-net architecture design for medical image segmentation through the lens of transformers. Medical Image Analysis, pp. 103280, 2024.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for seman- tic segmentation. Advances in neural information processing systems, 34:1 17644787577 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299 12990 2022.
Zhiheng Cheng, Qingyue Wei, Hongru Zhu, Yan Wang, Liangqiong Qu, Wei Shao, and Yuyin Zhou Un- leashing the potential of sam for medical adaptation via hierarchical decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3511-3522, 2024.
Dhakshinamoorthy Ganeshan, Phuong-Anh Thi Duong, Linda Probyn, Leon Lenchik, Tatum A McArthur, Michele Retrouvey, Emily Ghobadi, Stephane L Desouches, David Pastel, and Isaac R Francis. Struc- tured reporting in radiology. Academic radiology. 25(1):66-73, 2018.
Heng Guo, Jianfeng Zhang, Jiaxing Huang, Tony CW Mok, Dazhou Guo, Ke Yan, Le Lu, Dakai Jin, and Minfeng Xu. Towards a comprehensive, efficient and promptable anatomic structure segmentation model using 3d whole-body scans. arXiv preprint arXiv:2403, 240., 15063, 2024.
11

---

Published as a conference paper ICLR 2025
Ibrahim Ethem Hamamei, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Do- gan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Simsar, Mehmet Simsar, et al. A foundation model utilizing chest volumes and radiology reports for supervised-leve zero-shot detection of abnor- malities. arXiv preprint 2an::2203..77444 2024.
Nicholas Heller, Fabian Isensee, Dasha Trofimova, Resha Tejpaul, Zhongchen Zhao, Huai Chen, Lisheng Wang, Alex Golts, Daniel Khapun, Daniel Shats, et al. The kits21 challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-ppe ct. arXiv preprint arXiv:2307.019K. 2023.
Xinrong Hu, Xiaowei Xu, and Yiyu Shi. How to efficiently adapt large segmentation model (sam) to medical images. arXiv preprint arXiv:2306. 13731, 2023.
Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for lahel-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3942-3951, 2021.
Xiaoshuang Huang, Hongxiang Li, Meng Cao, Long Chen, Chenyu You, and Dong An. Cross-modal condi- tioned reconstruction for language-guided medical image segmentation. arXiv preprint arXiv:2404.02844 2024a.
Yuhao Huang, Xin Yang, Lian Liu, Han Zhou, Ao Chang, Xinrui Zhou, Rusi Chen, Junxuan Yu, Jiongquan Chen, Chaoyu Chen, al. Segment anything model for medical images? Medical Image Analysis, 92: 103061, 2024b.
Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus Maier-Hein. nnu-net: a self- configuring method for deep learning-base bbsed biomedical image segmentation. Nature methods, 18(2): 203-211, 2021.
Yankai Jiang, Zhongzhen Huang, Rongzhao Zhang, Xiaofan Zhang, and Shaoting Zhang, Zept: Zero-shot pan-tumor segmentation via query duery-disentangling and self-prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11386-11397, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Haoran Lai, Qingsong Yao, Zihang Jiang, Rongsheng Wang, Zhiyang He, Xiaodong Tao, and S Kevin Zhou. Carzero: Cross-attention alignment for radiology zero-shot classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11137-11146, 2024.
Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Unielip: Unified framework for contrastive lanuage-image pre-training. Advances in Neural Infor- mation Processing Systems, 35:1 15:1009-101,, 2022.
Zihan Li. Yunxiang Li, Qingde Li, Puyang Wang. Dazhou Guo. Le Lu, Dakai Jin, You Zhang, and Qingqi Hong. Lvit: language meets vision transformer in medical image segmentation. IEFE transactions on medical imaging, 2023.
Jingyang Lin, Yingda Xia, Jianpeng Zhang, Ke Yan, Le Lu, Jiebo Luo, and Ling Zhang. Ct-glip: 3d grounded language-imag image pretraining with ct seans and radiology reports for full-body scenarios. arXiv preprint a40iv:2404. 15272, 2024.
12

---

Published as a conference paper ICLR 2025
Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21152- 21164, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arxiv: 2017.
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1))654, 2024.
J Martijn Nobel, Koos van Geel, and Simon GF Robben. Structured reporting in radiology: systematic review to explore its potential. European radiology, pp. 1-18, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: arXiv:1807.0000 1807.03748, 2018.
Vu Minh Hieu Phan, Yutong Xie, Yuankai Qi, Lingqiao Liu, Liyang Liu, Bowen Zhang, Zhibin Liao, Qi Wu, Minh-Son To, and Johan W Verjans. Decomposing disease descriptions for enhanced pathology detection: pre-training framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11492 11501, 2024.
A multi-aspect
Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards building multilingual language model for medicine. Nature Communications, 15 (1):8384, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763 PMLR, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00 00714, 2024.
Tal Shaharabany and Lior Wolf. Zero-shot medical image segmentation based on sparse prompt using finetuned sam. In Medical Imaging with Deep Learning, 2024.
Chuyun Shen, Wenhao Li, Yuhang Shi, and Xiangfeng Wang. Interactive 3d medical image segmentation with sam 2. arXiv preprint arXiv:2408. 02635, 2024.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 20730-20740, 2022.
Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert-level de- tection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomed- ical Engineering, 6 6(19) 1101:19999-460.....11111 2022.
13

---

Published as a conference paper ICLR 2025
Tassilo Wald, Saikat Roy, Gregor Koehler, Nico Disch, Maximilian Rouven Rokuss, Julius Holzschuh, David Zimmerer, and Klaus Maier-Hein. Sam. md: Zero-shot medical image segmentation capabilities of the segment anything model. In Medical Imaging with Deep Learning, short paper track, 2023.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre- pre-training for x-ray diagnosis. In Proceedings the the IEEE/CVF International Conference on Computer Vision, pp. 21372-21383, 2023.
Yosuke Yamagishi, Shouhei Hanaoka, Tomohiro Kikuchi, Takahiro Nakao, Yuta Nakamura, Yukihiro No- mura, Soichiro Miki. Takeharu Yoshikawa, and Osamu Abe. Zero-shot 3d segmentation of abdominal organs in ct scans using segment anything model 2: Adapting video tracking capabilities for 3d medical imaging. arXiv preprint arXiv:2408.c S.Kiv:2408.0617000 2024.
Yiwen Ye, Yutong Xie, Jianpeng Zhang, Ziyang Chen, and Yong Xia. Uniseg: prompt-driven universal segmentation model as well as strong representation learner. In International Conference Medical Image Computing and Conputer-Assisted Intervention, pp. 508-518. Springer, 2023.
Mingze Yuan, Yingda Xia, Hexin Dong, Zifan Chen, Jiawen Yao, Mingyan Qiu, Ke Yan, Xiaoli Yin, Yu Shi, Xin Chen, et al. Devil is in the queries: advancing mask transformers for real-world medical image seg- mentation and dittr--disttibutionn localization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 23879-23889, 2023.
Kaidong Zhang and Dong Liu. Customized segment anything model for medical image segmentation. arXiv preprint arXiv:2304. 13785, 2023.
Lian Zhang. Zhengliang Liu, Lu Zhang, Zihao Wu, Xiaowei Yu, Jason Holmes, Hongying Feng, Haixing Dai, Xiang Li, Quanzheng Li, et al. Segment anything model (sam) for radiation oncology. arXiv preprint arXiv:2306. 2023.
Xuzhe Zhang, Yuhao Wu, Elsa Angelini, Ang Li, Jia Guo, Jerod M Rasmussen, Thomas O'Connor, Pathik D Wadhwa, Andrea Parolin Jackowski, Hai Li, et al. Mapseg: Unified unsupervised domain adaptation for heterogeneous medical image segmentation based on 3d masked autoencoding and pseudo- labeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5851-5862, 2024.
Jiayuan Zhu, Yunli Qi, and Junde Wu. Medical sam Segment medical images as video via segment anything model 2. arXiv preprint arXiv:2408.008777 2024.
14

---

Published as a conference paper ICLR 2025
A PRELIMINARIES
Problem Formulation of Zero-Shot Lesion Segmentation. Traditional semantic segmentation is inher- ently limited a closed-set setting and often struggles in real-world applications where there can be anoma- lous categories in the test data unseen during training. In contrast, the goal of zero-shot segmentation is segment objects belonging to categories that have not been encountered during training. In this work, we explore the zero-shot setting for lesion segmentation in 3D CT images. Specifically, there are two non- overlapping foreground sets: (I) N seen (base) categories of lesions denoted as C*; (2) No unseen (novel) categories of lesions denoted as C C* nC" 2. The training data is constructed from the images and la- bels that contain any of the N seen categories. Among these images, there may also exist unseen categories from C whose annotations can not be accessed during the training. Malenia aims to segment lesions, both seen and unseen, in the test data.
Recently, transformer-basee segmentation model Fask2Former (Cheng al. 2022 that can generate a set of segment-level embeddings have been applied and achieved promising performance in the field of medical image segmentation (Yuan al. 2023 Chen al.) 2023 Jiang al. 2024). Different from standard segmentation methods, Mask2Former adopt an additional lightweight transformer decoder along with a vision encoder and a pixel decoder. By feeding N mask tokens (also called segment queries) and pixel decoder features into the transformer decoder, Mask2Former generates N segment-level embeddings with masked attention layers, effectively partitioning an image into N corresponding regions. The final binary mask segmentations are obtained by computing the dot product of the mask tokens with the image features from the last pixel decoder layer, while an additional MLP predicts the class for each mask. In this work, extend the closed-set setting of mask classification an open-set setting through fine- ane-grrieed mask-text alignment. Specifically, we propose enhancing and matching mask tokens with text embeddings of the visual attributes of pathologies within multi-scale transformer decoder blocks. Additionally, we leverage the complementary strengths of visual and text representations for mask refinement, demonstrating strong generalizability and improved zero-shot lesion segmentation performance.
Mask Tokens.
B DATASET DETAILS AND DATA PREPROCESSING
B.1 DATASET DETAILS
Our study utilizes annotated datasets encompassing lesions across 12 diseases and 6 organs, derived from both public and private sources. The private datasets particularly include paired CT scans and radiological reports providing systematic imaging descriptions of lesions. We summarize all the datasets in Table 5
Public Datasets:
Kidney Tumor and Kidney Cyst. This dataset is part of the Kidney and Kidney Tumor Segmentation Challenge (KiTS23) (Heller al. 2023), ; which provides 489 cases of data with annotations for the segmentation of kidneys, renal tumors, and cysts.
Liver Tumor, Pancreas Tumor, Pancreas Cyst, Colon Tumor, and Lung Tumor. These datasets are part of the Medical Segmentation Decathlon (MSD) (Antonelli et al. 22022), providing annotated datasets for various lesions. Notably, Pancreas Tumors and Cysts are grouped under a single region of interest in the MSD dataset but were separately classified by experienced radiologists as part of this study.
Private Datasets:
For lesions underrepresented in public datasets, we utilized private datasets annotated by radiol- ogists. These datasets include lesions such as kidney stones, liver cysts, gallbladder cancer, and
15

---

Published as conference paper ICLR 2025
Table 5: Details of Datasets. denotes MSD-Pancrees is reannotated to pancreas tumor and pancreas cyst.
Dataset Name
Segmentation Lesion Targets
of scans
KiTS23
Kidney Tumor, Kidney Cyst
489
MSD-Colon Tumor
Colon Tumor
126
MSD-Liver Tumor
Liver Tumor
131
MSD- MSD-Hepatic Vessel Tumor
Hepatic Vessel Tumor
303
MSD-Lung Tumor
Lung Tumor
64
MSD-PPancreas
Pancreas Tumor
216
MISD-Pancreas
Pancreas Cyst
65
Liver Cyst
30
Gallbladder Tumor
30
Private Data
Gallstones
30
Kidney Stone
30
gallstones, accompanied by paired radiological reports. Each dataset consists of 30 cases, provid- ing balanced sample size for preliminary analysis.
B.2 CREATING STRUCTURED DISEASE ATTRIBUTE DESCRIPTIONS
Specifically, For the public datasets MSD and KiTS23, which only provide mask annotations, two senior diologists supplemented the annotated lesions with diagnostic reports and corresponding lesion descriptions of the eight visual attribute aspects. For the in-house data, utilized the semi-automatic annotation pipeline described in Sec First, we prompted GPT-4 (Achiam et al.] 2023) to extract lesion-related descriptions from the findings section of each report. Then, experienced radiologists collaborated review, cor- rect, supplement, and expand the GPT-generated visual attribute descriptions. Through this process, we established eight standardized attributes for describing lesions, which form the basis our structured re- ports. These attributes provide comprehensive and objective template for characterizing lesions, ensuring consistency and enabling generalization across different types of lesions, as shown Table
B.3 DATA PREPROCESSING
We pre-process CT scans using isotropic spacing and uniformed intensity scale to reduce the domain gap among various datasets. We determine whether each CT scan is contrast-enhanced by evaluating the Hounsfield unit (HU) values of the aorta and inferior vena cava. the average HU both the aorta and inferior cava is less than 80, the is classified non-contrast. Otherwise, it is classified contrast- enhanced. This helps radiologists determine the attribute description "Enhancement Status".
B.4 JUSTIFICATION FOR THE LEGALITY DATA USE
This study utilizes data from both public and private sources. (1) The public data is obtained from the MSD (Antonelli al., 2022) and KiTS23 (Heller et al. 2023) datasets, which are publicly available and provide CT images along with lesion masks. We have supplemented these lesion masks with corresponding structured lesion attribute descriptions. Since these datasets are fully open and no patient privacy information was disclosed during their use, data usage is fully justifiable and legal. (2) The private dataset is sourced from the First Hospital China Medical University. This retrospective study was approved by the Ethics Committee of the First Hospital of China Medical University, and the informed consent requirement waived. The private dataset includes CT images, lesion masks, and diagnostic reports. All diagnostic reports in this dataset have been fully anonymized, with all patient-identifaale identifiable information and private details re-
16

---

Published as conference paper ICLR 2025
Table 6: Definitions and detailed content of the eight disease attributes.
Attribute
Definition
Content
Enhancement Status
Intravenous contrast agent usage
Ennanncdd CT", "oo."" "Non-contrast CT
"Colon", "Liver", "Pancreas" "Right Lung". "Left Lung" "Right Kidney", "Left Kidney", "Gallbladder"" "Hepatic Vessel" "Round-like"" "Irregular".
Location
Organ-specific anatomical regions
"Wall thickening", "Punctate".
Morphological characteristics
Shape
the lesion
"Nodular", "Cystic", "L "Luminal narrowing".
"Protrusion into the lumen"
Relationship Surrounding Organs
Invaston of or proximity udjacent organs
close relationship with surrounding organs".
"Close relationship with adjucent organs"
"Hypodense lesion", "Isodense lesion". "Hyperdense lesion". "Mixed-density lesion", "Hypodense fluid-like lesion". "I "Isodense soft tissue mass", "Low-density ground-gaas opacity"
Radiographic attenuation properties of the lesion
Density
Uniformity of attenuation within lesion
Density Variations
"Homogeneous", "Heterogeneous"
Well-defined margin"..
Features of the lesion's border and surface texture
"Clear serosal surface"..
Surface Characteristics
"Ill-defined margin".
"Serosal surface irregulurity"
Distinctive attrihutes indicating lesion characteristics
"Spiculated margins"
Specific Features
"Retention of pancreatic fluid".
moved. For all anonymized diagnostic reports, we exclusively utilized the GPT-4 API (Achiam et al. 2023) )
process textual descriptions of lesions, converting them into structured format. No patient-identifaabl information was disclosed throughout the data processing, ensuring that our data usage is completely justi- fiable and legal.
C FINE-TUNING EVALUATION
Malenia can also serve as a valuable pre-training method for downstream segmentation tasks a fine- tuning setting. To evaluate this, adopt various vision-language pre-training strategies to pre-train the 3D image encoder of a nnUNet model using our training data. We then fine-tune the nn on downstream datasets using the pre-trained weights. For all downstream datasets, we split each into 50% for training, 20% for validation, and 30% for testing. As shown in Table 7 Malenia consistently outperforms previous methods, further validating the importance of tailoring vision-language pre-training for dense prediction tasks in the medical domain, where fine-grained understanding of disease-related features is essential. Notably, Malenia significantly surpasses the second-best method, CT-GLIP, by 6.03% and 3. 3.77 in DSC hepatic vessel tumor and pancreas cyst segmentation, respectively. On our in-house dataset, Malenia also delivers a substantial performance boost. The fine-tuning evaluations demonstrate that injecting fine- grained, lesion-level cross-modal knowledge from both the vision and language domains to enhance mask representations during pre-training improves the model's transferability, particularly for fine-grained lesion segmentation tasks.
17

---

Published as a conference paper ICLR 2025
Table 7: Segmentation results of different vision-language pretraining models under the fine-tuning setting.
MSD
K7T823
In-house Dataset
Method
Hoparic Vessel Tumor
Panoreas Cyst
Kidney Tumor
Liver
Cyst
Kidney Stone
Gallbladder Tumor
DSC1
NSD
DSCt
NSD
DSC1
NSD1
DSC1
NSD1
DSC1
NSD1
DSC
NSD
Net (from Scratch)
62.44
72.87
64.15
76.27
70.59
76.79
67.84
74.03
50.47
59.91
56.70
64.68
GLoRIA [Hunn 2021
63.11
73.02
64.55
76.41
72.03
77.82
68.35
74.87
50.82
60.25
57.53
65.46
BioViL
Brecking ale 2022
63.14
73.08
64.57
76.44
72.10
77.94
68.66
75.02
50.88
60.27
57.62
65.69
Mf el 2026
64,30
74.45
65.20
77.35
72.87
78.71
69.15
75.56
51.42
61.65
58.34
66.40
MAVL
of 2054
65.49
75.51
77.61
74.09
80.23
69:23
75.60
52.30
66.77
CT-CLIP Hlamane
2024
65.66
75.73
65.85
77.92
74.12
80.24
69.57
75.82
52.45
62.37
58,80
66.84
CT-GLIP
aL.
76.11
Malenia
71.96
81.83
69.95
82.00
76,88
82.55
72.44
79.84
53.35
63.69
64.21
72.63
Table 8: Ablation study on using different LLMs for attribute construction. As described in Sec. B.2 the LLMs are used only for our in-house dataset, as public datasets lack diagnostic reports and instead rely on attributes directly annotated by human experts.
In-house Dataset
Method
Liver Cyst
Kidney Stone
Gallbladder Tumor
DSC
NSD
DSCt
NSD
DSCt
NSD1
MMed-Llamas
54.97
63.06
38.66
47.71
43.28
51.45
GPT4
54.98
63.08
38.67
47.73
43.28
51.46
MMcd-Llama-3-28 + Human Annotators
61.84
70.93
43.03
52.94
47.34
55.79
GPT4 Human Annotators
61.85
70.93
43.05
52.95
47.35
55.79
D ADDITIONAL ABLATION STUDIES
D.I DIFFERENT LLMS FOR ATTRIBUTE CONSTRUCTION.
To transform patient reports into structured descriptions of fundamental disease attributes, we employ a semi-automatic pipeline that uses GPT-4 (Achiam et al. 2023) for extracting lesion-related descriptions from each report, followed by further refinement by experienced human annotators. Given that using GPT- 4 (Achiam al.) 2023) for attribute construction from radiological reports often impractical due to privacy and legal concerns, we conduct extensive experiments to determine whether switching to locally hosted open-source LLMs can achieve comparable performance. The results are shown in Table We compare the segmentation performance on unseen lesion categories using either the open-soure medical large language model MMed-Llama-3--3 (Qiu et 2024) or GPT-4 (Achiam et al. 2023) to extract attribute descriptions from radiological reports.
The results indicate that using GPT-4 (Achiam et al. 2023) or MMed-Llama-3-88 (Qiu et al. 2024) for attribute extraction from radiological reports, with or without human annotation, has negligible effect on model performance. This largely because extracting and structuring attributes already present in the re- ports is straightforward and casy task. However, incorporating human annotators significantly enhances performance, as some attributes are not explicitly mentioned in the reports and require expert supplementa- tion-a task beyond the capabilities of current large language models. Given the high cost of human anno- tation, and to support future research, we will open-source our attribute annotations for all public datasets used in this study, along with our code, benefit the research community.
18

---

Published as conference paper ICLR 2025
Hyper-parameter analysis the number of masik takens
54.96
52
48
4743
47.34
46
46.24
DSC Kidney Tumor on KITS23
42.62
42
12
20
24
28
Number Mask Tokers
Figure 4: Ablation studies segmentation performance of unseen lesions using different numbers of mask
tokens.
D.2 THE NUMBER OF MASK TOKENS
An important hyperparameter in our framework is the number of mask tokens. We conducted ablation experiments to assess the segmentation performance of unseen lesions using different numbers of mask tokens on the KiTS23 dataset (Heller al. 2023) and our in-house dataset. The results are presented in Fig. 4 We observed that when N is less than 12, the model's performance drops sharply. This observation is consistent with previous findings from MaskFormer-basee
methods (Cheng al. 2021 [2222 Jiang et al., 2024). which suggest that the number of queries should be greater than the number of possible useful classes in the data. When N exceeds 16, no significant performance improvement was observed. Therefore, we set the default number of mask tokens to N 16.
D.3 THE NUMBER OF ATTRIBUTE ASPECTS
This section addresses the key question of whether adding more attribute aspects improves zero-shot seg- mentation performance. We conducted ablation experiments evaluate the segmentation performance of unseen lesions we gradually added attribute categories in the order defined in Table The results are presented Fig. We observed that as the number attribute categories used for vision-language align- ment increased from to the model's is zero-shot performance consistently improved with greater attribute diversity. This result demonstrates the superiority our approach in using multiple attribute aspects to de- scribe lesions from different perspectives, providing valuable textual features that help capture subtle visual characteristics of the lesions. This finding also suggests that incorporating additional contextual information advanced features enhance the model's ability to capture complex visual semantics.
19

---

Published as conference paper ICLR 2025
Ablation study of adding higher numbers attributes by order.
DSC of Kidney Tureer KITS23
Gallbladder Tumor in-house Dataset
$2.38
53.75
54.96
50
47.04
49.09
50.33
46.41
45.88
43.58
40
33.96
35.02
30
5
Number of added altribute callegories
Figure 5: Ablation studies segmentation performance of unseen lesions using different numbers of at- tribute aspects.
Table 9: Zero-shot lesion-attribute matching performance (%) of Malenia.
KiTS23
In-house Dataset
Attribute Aspect
Hepatic Vessel Tumor
Pancreas Cyst
Kidney Tumor
Liver Cyst
Precisiont
Recallf
Precision?
Recall?
Precision
Recal!?
Precision!
Recall
Enhancement Status
100
100
100
100
100
100
100
100
99.67
Shape
93.71
97.03
89.23
90.77
94.27
93.67
86.67
90.00
Kelationship with Surrourding Organs
99.01
98.33
96.92
98.45
98.18
98.37
93.34
96.67
Density
92.41
93.40
92.31
93.76
94.35
94.87
92.65
92.11
Densily Variations
95.62
96.13
93.04
92.81
94.52
93.79
90000
91.34
Surface Characterisies
91.62
89.83
90.45
92.44
88.05
Specilic Features
84.54
85.46
82.13
82.40
83.29
82.96
82.75
83.34
EVALUATION OF LESION-ATTRIBUL MATCHING
Mask-attribute alignment is the core foundation of our method. In this section, we report the zero-shot axsion-attribute matching performance of Malenia. Table/9presents the lesion-level attribu matching precision and recall for each attribute aspect the zero-shot testing dataset. Malenia achieved outstanding zero-shot performance in the task lesion-atrribute alignment. Notably, we observed that attributes that relatively easier determine, such "Enhancement Status" and "Location", the model attained nearly 100% precision and recall. However, for attributes requiring more complex visual semantic understanding, such "Surface Characteristics" "Density", and "Specific Features" there is still room for improvement. This may represent a meaningful research direction for future work. In Fig. 6 we present visualizations of some of Malenia's segmentation and attribute alignment results.
20

---

Published as conference paper ICLR 2025
CT Image
Ground Truth
Malenia
CT Image
Ground Trush
Malrnia
Location:
"Live"
Location:
"Parxicaa"
Shapls
Shape
"Bamddlike"
Density:
"Hypedonse Lesien"
0.57
Densh:::
"Hypolaree Lesis""
6:22
Demsty
"Hatengennous"
"Homogeseous"
Enhenoeeer States
"Irbanccd CI'
Echamemest Stouss
"Entanced CT"
Chanoertittes
"Ill Margin"
5,7
Surfice Chractoiitios
Margin"
Summunding Organs:
sdattessipp with
"No Close Reittinship with Adjasnt Orpine"
Relatonship with
Sumunding Orgars:
"Close Relzionship with Adjaces Organs"
0.55
Sperticc
"Prencru Devraseel Danity
Spacifi: Festunes
Dansiyy
CT Image
Ground Trubh
Malenia
CT Image
Ground Truth
Malenia
Foration:
"Talen
12odion:
"Right lang"
"Insgudar Thickening"
"Erngular"
Denaly:
"Hypendasse Lesien"
Dessity:
"Low Densty Ground Glass Opacity"
oas
Vadition:
Densty Variation:
Enhancement Statu:
"Echanced CT
Frhunoment State:
"Non-Cuntnal CTP"
Margin"
Surface Clarateristics
"Ell Defined Margii"
8.12
Summnndnng Drgans:
Relatonship with
Rehtiunship with Adjucan Organe" 0.6
Surounding Oagane:
Relkionship with
"No Class Relatisnehip with Adjarami Organs"
137
Specliic Tesiune:
Incressed Density
Specttic Postures
"Spisulated
Figure 6: Visualization segmentation and aattib-e matching results of Malenia. The numbers indicate the normalized similarity scores between the lesion masks and their corresponding attributes.
21

---

Published as conference paper ICLR 2025
Jaun-la Iaap
Gread Tru
Redh
TE:
MT:
Zeorr-lia Iingge
Genural Trais
Bech
SIT.
Figure 7: Visualization illustrating how utilizing both mask and text embeddings improves segmentation performance for unseen and seen lesions with ambiguous boundaries. We present several prediction cases generated using both text embeddings and mask tokens, text embeddings only (TE), and mask tokens only (MT).
F ANALYSIS OF MODEL ROBUSTNESS
F.1 HANDLING LESIONS WITH AMBIGUOUS BOUNDARIES
Fig. 7 provides detailed examples illustrating how the CMKI module integrates text tokens and mask tokens enhance the performance for lesions with ambiguous boundaries. We observed that when lesion bound- aries are particularly blurry (as seen in the Hepatic Vessel Tumor cases on the left side of Fig. 7 the visual features from the mask tokens fail to accurately capture the lesion's area. such instances, text descriptions of attributes-suce segmentation generated by the corresponding text embeddings refines the results produced by mask tokens alone, leading improved performance.
the lesion's 's shape and surface characteristies-Pree provide additional information. The
Additionally, when encountering lesions with highly atypical appearances, such as colon tumors, which exhibit significant variability in shape and size well as poorly defined boundaries (as seen in the colon tumor cases on the right side of Fig. 77. we observed that the model tends to produce more false positives when relying solely on mask tokens for segmentation predictions. Due the clear semantic information and specific context provided by distinct attribute aspects, such as density, density variations, and surface texture, segmentation results generated using attribute embeddings exhibit significantly fewer false positives. These examples demonstrate how our framework leverages the strengths of text embeddings and mask tokens to address ambiguous or borderline cases, such as lesions with poorly defined boundaries or highly atypical appearances.
F.2 HANDLING INCORRECT TEXT INPUTS
During inference, we use all stored attribute embeddings. However, users may wish to specify their own attributes during testing, which could lead to the inclusion of incorrect attributes. In this section, we provide an example to illustrate how our model handles ambiguous or incorrect text inputs in Fig. 8 Specifically, we present two scenarios.
(1) The model has access to both correct and incorrect text inputs (Fig. (a). In this case, Malenia continues to produce stable segmentation results. Owing to our proposed aask-attribute alignment strategy, Malenia accurately computes the similarity between different text input embeddings and the mask tokens, effectively filtering out incorrect text inputs and selecting the best matched attributes. (2) The model only has access to incorrect text inputs from users (b). For example, the image may contain lung tumor in the right lung, but the provides attribute for the left lung. In this case, visualized both the segmentation heatmap generated by the text embeddings and the one generated by the mask tokens. As shown in (Fig. (8),, although the text inputs incorrect, the similarity between these inputs and the
22

---

Published as a conference paper ICLR 2025
Table 10: Computational cost comparison between Malenia and competing methods used in our evaluation experiments. The FLOPs are computed based input with spatial size of 96 96 x 96 the same A100 GPU.
Method
Malenia
ZePT
I1-SAM
nnU-Net
Swin UNETR
Efficiency
Params
323, 36.
743.94M
557.60M
370.74M
371.94M
FLOPs
1128.960
1337.59G
3643,960
6742.36G
2005.48G
image features remains very low, resulting in segmentation heatmaps with low confidence. contrast, the mask tokens continue to produce accurate segmentation heatmaps with high confidence. Thus, during the ensembling process, the high-confidence heatmaps from mask tokens prevent Malenia from being influenced by incorrect text inputs.
G COMPUTATIONAL COST COMPARISON
We compare the number of parameters across different methods to evaluate computational efficiency. More- over, we also assessed the inference speed of various models, as this crucial for clinical applications. We utilized floating- ppinn operations per second (FLOPs) as metric to measure the inference speed. Table presents the number parameters and FLOPSs (G) for Malenia and the comparison models, demonstrating that our method not only achieves the best performance but also superior computational efficiency. Re- markably, due to the mask-attribute alignment mechanism, Malenia directly matches mask embeddings with stored attribute embeddings and does not require additional text encoder process text inputs dur- ing inference. This makes it more flexible and efficient than Sanguage-drive segmentation models like ZePT (Jiang et al. 2024), which rely user-provided text prompts based the input images. Additionally, Malenia does not require an extra encoder to process complex visual prompts. Instead, it directly generates segmentation predictions for the input image while also producing attribute descriptions. This approach is more aligned with real clinical diagnostic scenarios, where doctors expect the Al model to first provide diagnostic predictions assist in the diagnostic process.
H DISCUSSION ON DIFFERENCES BETWEEN ZEPT AND MALENIA
The primary distinctions between approach and ZePT (Jiang et al. 2024 are as follows:
Alignment Strategy. ZePT Jiang et al. (202)) uses a single-scale mask-tex. alignment, aligning only the image features from the final Transformer decoder block with basic textual description. In contrast, our method employs multi-scale mask-attribute alignment, aligning multi-scale image features with detailed lesion attribute descriptions through multi-positin contrastive loss. This represents a key technical advancement of our approach. The alignment strategy used by ZePT misses the opportunity leverage multi-scale features, which hinders its ability capture targets with significant scale variations. Furthermore, ZePT treats each mask and its corresponding cate- gory as the only positive sample pair, resulting in weak associations between different categories within the learned feature space. For instance, in ZePT, the mask corresponding to a liver tumor and the text embeddings of liver cyst are treated as negative sample pairs, despite the shared attributes between liver tumors and liver cysts. Such similarities are overlooked by ZePT, whereas in our approach, these similarities are explicitly captured through attribute alignment. Therefore, Malenia demonstrates stronger generalization capabilities.
Mask Prediction. ZePT et al. (2024) relies solely on mask tokens for segmentation mask pre- diction. In contrast, our approach introduces a Cross- Modal Knowledge Injection (CMKI) module,
23

---

Published as conference paper ICLR 2025
CT Image
Ground Truth
Malenia
"Left Lung"
"Right Lung"
Incorrect Text of "Location"
Correct Text of "Location"
Similarity Score: 0.01
Similarity Score: 0.99
Discarded!
Selected!
(a) Both correct and incorrect attribute texts are provided
"Left Lung"
Pixel A ctivations From Incorrect Text Tokens
Pixel A ctivations From Mask Tokens
Segmentation Prediction
(b) Only incorrect attribute texts are provided
Figure 8: Visualization of how Malenia handles incorrect text inputs.
24

---

Published as a conference paper ICLR 2025
Table 11: Details of the clinical knowledge table used during inference. We present several common diseases along with their corresponding attributes. We will open-source all diseases involved in this study along with their corresponding attributes.
and Heprrmie
Tour
Iay leae 13y
Hypakeer
Kiay Rew Raka
Nokda
pupdde loke
No das clacedly
Hqpestios
Voogl Tiees
Sregers Briegrass
Well Sieen restoo RE
Piou snreeniiion fiw snt Pioeeee rersodd Saary
Bropaan
beed Bravey
Willer TE
which integrates features from both vision and language modalities, utilizing both text tokens and mask tokens for segmentation mask prediction. We demonstrate in Fig. 7 that CMKI is particu- larly effective in zero-shot lesion segmentation tasks, where unseen categories can be recognized by leveraging the complementary strengths of both vision and language modalities.
Thus, the main technical contributions of our work are: (1) Leveraging multi-scale features for cross-modal alignment. (2) Decomposing textual descriptions/repprts into categorized attributes. (3) Introducing multi- positive alignment mechanism that establishes better cross-modal feature representations. (4) Developing the Cross-Modaa Knowledge Injection module. Our ablation studies also demonstrate that incorporating these designs significantly enhances zero-shot generalization performance. Furthermore, Fig. shows that leveraging fine- -reaggrreed textual information facilitates tumor segmentation, especially in scenarios where visual cues are limited.
I LIMITATIONS AND FUTURE WORK
Malenia currently demonstrates superior zero-shot lesion segmentation performance, nearing the perfor- mance of fully supervised, task-specific models. However, several limitations persist, revealing valuable directions for future research:
Cross-domain and cross-modality challenges: Zero-shot lesion segmentation across anatomical re- gions and imaging modalities remains highly challenging. For instance, training the model on lesions in abdominal CT data and directly testing on brain cancer MRI is challenging due significant differences in anatomical structures and feature distributions between the training and testing images. Achieving robust cross-domain and modalityy zero-shot generalization is a complex but essential research direction in both natural and medical imaging domains.
Scalability to other imaging modalities: This work focuses primarily on 3D CT scans. While radiology reports for other modalities, such as X-ray and MRI, also contain structured attributes that could be leveraged, Malenia's 's applicability to these modalities has not yet been tested. Future work will focus on adapting the framework to handle these imaging modalities, addressing their unique characteristics such spatial resolution and semantic differences.
Handling lesions with ambiguous boundaries: Defining the boundaries of lesions with unclear or diffuse margins, such as pancreatic cancer or colon cancer, is inherently challenging and subjective even for radiologists. Our analysis in Table 9 indicates that Malenia's 's mask-attribute matching performance on attributes such as Surface Characteristics still has room for improvement, as the inherent uncertainty in the corresponding ground truth labels affects segmentation accuracy. Future efforts will focus on enhancing the model's ability to handle these ambiguous cases, potentially incorporating additional contextual information.
25

---

Published as conference paper ICLR 2025
Capturing complex visual semantics: Malenia performs less effectively on attributes requiring in- tricate visual semantic understanding, such as Surface Characteristics and Specific Features. The challenges here may arise from the variability and imbalance in specific feature distributions, as well as the inherent difficulty in categorizing rare or atypical features. Exploring advanced model architectures and integrating additional contextual information will be crucial for enhancing per- formance on these attributes.
Generalization to diverse lesion types: While Malenia demonstrates strong performance on 12 lesion categories, the diversity of lesions in clinical practice is vast. Future work will focus on eval- uating Malenia on a broader range of lesion types to further validate its generalization capabilities and ensure applicability diverse clinical scenarios.
We believe addressing these limitations will significantly enhance Malenia's robustness, scalability, and clinical utility, contributing to the broader advancement of zero-shot learning in medical imaging.
26


Reviews:
[{"review_id": "IcNOQGAYIG", "reviewer": "Reviewer_Lt5s", "summary": "This work presents a zero-shot medical image semantic segmentation framework that is based on maskformer and attribute-based visual-language feature fusion. To achieve fine-grain alignment between visual and textual concepts, the authors propose to decompose radiological reports into attributes describing lesions of interest, assisted by LLMs and human annotators. The obtained attributes allows fine-grain fusion and alignment between lesion features and textual concepts, yielding improved generalization performance upon unseen lesions. The proposed framework is evaluated on tumor segmentation tasks, and demonstrates improved segmentation results compared with existing zero-shot segmentation approaches on unseen lesions.", "questions": "Considering the practicality issue, would replacing GPT-4 with an open-source LLM for attribute construction (probably also without human annotators?) reaching a similar level of performance gain (as mentioned above)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of decomposing free-text reports into attributes for fine-grain training and inference is intuitive and feasible. It can significantly reduce the noise and ambiguity associated with unstructured free-text reports and yields improved performance on unseen lesions.\n\nThe key components and the detailed implementations are described in good detail. \n\nThe proposed approach demonstrates performance improvements compared with previous zero-shot image segmentation methods.\n\nDetailed ablation studies are presented to verify the effectiveness of key components.", "weaknesses": "Despite stronger segmentation performance, the major framework looks similar to ZePT (Jiang et al., 2024): maskformer backbone + interacting visual and textual features that are obtained from detailed description of the lesions. Therefore, the key take-home message for readers may be a bit unclear: should the readers interpret the key technical contribution as explicitly decomposing textual descriptions/reports into categorized attributes for the ease of learning and inference?\n\nReaders may argue that using GPT-4 for attribute construction from radiological reports is often infeasible in practice due to the privacy and legal concerns. The authors are encouraged to argue if switching to local open-source LLMs would yield a similar level of performance gain. \n\nThe authors may want to decompose some of technical details in Figure 1 into separate figures and move them closer to corresponding paragraphs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wtvOx6FE3Z", "reviewer": "Reviewer_thtg", "summary": "This paper introduces Malenia, a framework for zero-shot lesion segmentation in 3D medical images, aimed at addressing the challenge of transferring image-level knowledge to pixel-level segmentation tasks. Building on advancements in medical vision-language pre-training, Malenia integrates a multi-scale mask-attribute alignment framework and a Cross-Modal Knowledge Injection module to link visual and textual features, enabling the model to handle previously unseen lesions. The authors evaluate Malenia's performance through experiments on three datasets (two public and one private) across 12 lesion categories.", "questions": "please refer to Weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The topic on zero-shot lesion segmentation is valuable, as clinical settings often involve diverse, emerging anomalies, and data collection is challenging, increasing the need for models that handle unseen diseases in an open-set context.\n2. The authors strengthen their claims with both qualitative and quantitative results, enhancing the credibility of their findings and providing a well-rounded evaluation of the proposed approach.\n3. The paper is well-organized, with a clear and logical structure that makes complex concepts accessible and easy to follow for readers.", "weaknesses": "1. The Zero-Shot Inference section lacks sufficient detail on how the model handles test CT images containing unseen tumor types. For instance, if a test image includes both a kidney tumor and a gallbladder tumor, it is unclear how the model predicts these different tumors within the same image. Additionally, the phrase “we obtain the class information for each predicted lesion mask by referencing a clinical knowledge table” is ambiguous. Further clarification is needed on what this clinical knowledge table entails and how it is used in the inference process.\n2. The comparisons in Table 1 and Table 2 may not be entirely fair. The authors introduce additional ATTRIBUTE DESCRIPTIONS annotations, which provide extra information not available to baseline methods like TransUNet, nnUNet, and Swin UNETR. These baselines rely solely on image data, while the proposed approach leverages attribute annotations, giving it an advantage in performance comparisons.\n3. The fully-supervised performance of baseline methods reported in Table 2 appears unusually low. For instance, in the official nnUNet paper, liver tumor segmentation achieved a Dice score of 76 on the test set, whereas the presented result here is only 61.33. Similarly, lung tumor segmentation originally reported a Dice score of 74, but this paper reports 54.44. This discrepancy raises concerns about the reproducibility and fairness of the comparisons.\n4. In Table 7, the authors compare their method to existing vision-language pretraining strategies, but it is unclear how these methods were reproduced. Most vision-language pretraining approaches require access to diagnostic reports, which are not included in the public MSD and KiTS23 datasets. It would be helpful to understand how the authors conducted these comparisons without complete diagnostic reports. Additionally, the absence of comparisons with important pretraining models like CT-CLIP weakens the evaluation.\n5. Scalability of the proposed method is questionable due to its reliance on ATTRIBUTE DESCRIPTIONS. For practical clinical applications, this approach would require physicians to provide additional lesion descriptions across eight visual attribute aspects, which may not be feasible in all settings. This dependency could limit the method’s applicability in diverse clinical environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LRHUHEeBCJ", "reviewer": "Reviewer_DS2M", "summary": "The paper introduces \"Malenia,\" a framework for zero-shot 3D lesion segmentation using vision-language pre-training methods. This framework aligns multi-scale mask representations of lesions with textual embeddings of disease attributes, which facilitates the segmentation of unseen lesion types in medical imaging.", "questions": "1. Could the authors provide insights on the computational efficiency of their framework?\n2. This work and its motivation are similar to the previous works [1-2], please describe the differences between them.\n[1] Li, Z., Li, Y., Li, Q., Wang, P., Guo, D., Lu, L., ... & Hong, Q. (2023). Lvit: language meets vision transformer in medical image segmentation. IEEE transactions on medical imaging.\n[2] Huang, X., Li, H., Cao, M., Chen, L., You, C., & An, D. (2024). Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation. arXiv preprint arXiv:2404.02845.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This study bridges the gap between mask representations and textual disease attributes for zero-shot lesion segmentation.\n2. The empirical validation includes comprehensive experiments across multiple datasets and lesion types, demonstrating the robustness and effectiveness of Malenia.", "weaknesses": "1. While the paper effectively addresses 3D CT scans, it does not discuss the applicability of the Malenia framework to other imaging modalities like X-ray.\n2. The paper does not discuss how the framework handles ambiguous or borderline cases where lesion boundaries are not well-defined or where lesions exhibit highly atypical appearances.\n3. There is no thorough discussion on the limitations or failures of the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I0op2jhQlj", "reviewer": "Reviewer_eodD", "summary": "The paper introduces Malenia, a novel framework specifically designed for 3D zero-shot lesion segmentation in medical images. The authors aim to address the challenges of transferring image-level knowledge to pixel-level tasks—such as lesion segmentation—by proposing a multi-scale lesion-level mask-attribute alignment approach. Malenia improves the alignment between the visual features of unseen lesions with textual representations, leveraging Cross-Modal Knowledge Injection (CMKI) to enhance both visual and textual embeddings. Experimental results across three datasets demonstrate that Malenia outperforms state-of-the-art (SOTA) methods in zero-shot lesion segmentation.", "questions": "1. In Section 3.1, the authors employ the Multi-Positive InfoNCE (MP-NCE) loss for aligning mask embeddings with multiple attribute embeddings. Could the authors provide more details on how the MP-NCE loss is computed and how it facilitates learning with multiple positive pairs? Specifically, how are the positive and negative pairs defined in the context of multiple attributes?\n   Could the authors explain how this formulation effectively handles multiple positive pairs for each mask token?\n2. The model uses a fixed number of mask tokens ($N=16$) and defines $R=8$ attribute aspects. Have the authors conducted any experiments to assess how varying $N$ or the number of attributes affects the model's performance? An ablation study on these hyperparameters could provide insights into the model's sensitivity and optimal settings.\n3. As noted in Appendix D, the model performs less well on attributes like \"Surface Characteristics\" and \"Specific Features\". What are the potential reasons for this lower performance? Could incorporating additional contextual information or advanced features help the model better capture complex visual semantics?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed Malenia framework is novel, particularly in its use of multi-scale mask-attribute alignment for 3D lesion segmentation. This is a meaningful extension of zero-shot segmentation methods to the medical domain, where unseen lesion categories are common (e.g., \"hepatocellular carcinoma,\" mentioned in Sec. 1).\n2. The CMKI module is a strong contribution because it combines visual and textual embeddings, enriching the segmentation prediction with complementary information from both modalities. This is a novel approach that enhances the zero-shot capability of the model.\n3. The paper introduces a multi-positive contrastive loss to fine-tune the alignment between lesion masks and textual attributes. This approach allows the model to generalize better to unseen lesions by learning from shared visual attributes, which distinguishes it from prior works like SAM (Kirillov et al., 2023).", "weaknesses": "1. The method relies on transforming patient reports into structured descriptions of eight visual attributes, which requires expert radiologist involvement. This process may not be scalable or practical in real-world applications where such annotations are not readily available.\n2. As noted in Appendix D, the model performs less effectively on attributes requiring complex visual semantic understanding, such as \"Surface Characteristics\" and \"Specific Features\". This suggests a potential limitation in capturing intricate visual patterns.\n3. While the model shows strong performance on 12 lesion categories, the diversity of lesions in clinical practice is vast. The ability of Malenia to generalize to a wider range of unseen lesions without additional training remains uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)