Segment as You Wish: Free-Form Language-Based Segmentation for Medical Images

## Abstract

Abstract Medical imaging is crucial for diagnosing a patient’s health condition, and accurate segmentation of these images is essential for isolating regions of interest to ensure precise diagnosis and treatment planning. Existing methods primarily rely on bounding boxes or point-based prompts, while few have explored text-related prompts, despite clinicians often describing their observations and instructions in natural language. To address this gap, we first propose a RAG-based free-form text prompt generator, that leverages the domain corpus to generate diverse and realistic descriptions. Then, we introduce FLanS , a novel medical image segmentation model that handles various free-form text prompts, including professional anatomy-informed queries, anatomy-agnostic position-driven queries, and anatomy-agnostic size-driven queries. Additionally, our model also incorporates a symmetry-aware canonicalization module to ensure consistent, accurate segmentations across varying scan orientations and reduce confusion between the anatomical position of an organ and its appearance in the scan. FLanS is trained on a large-scale dataset of over 100k medical images from 7 public datasets. Comprehensive experiments demonstrate the model’s superior language understanding and segmentation precision, along with a deep comprehension of the relationship between them, outperforming SOTA baselines on both in-domain and out-of-domain datasets.

## 1 Introduction

Medical imaging is crucial in healthcare, providing clinicians with the ability to visualize and assess anatomical structures for both diagnosis and treatment. Organ segmentation is vital for numerous clinical applications, including surgical planning and disease progression monitoring [, , ]. However, accurately segmenting organs and tissues from these medical images, i.e., medical image segmentation (MIS), remains a significant challenge due to the variability in patient positioning, imaging techniques, and anatomical structures [, ]. Recent advancements in large foundation models, such as Segment Anything Model (SAM) [] and MedSAM [], have shown promise in achieving more accurate and faster MIS. These models often require the users to input a predefined category name, a box, or a point as a prompt. However, in real-world scenarios, clinicians often rely on natural language commands to interact with medical images, such as “Highlight the right kidney” or “Segment the largest organ”. An accurate segmentation model with flexible text comprehension capability is therefore essential for a wide range of clinical applications.

The first challenge lies in the development of a segmentation model that can handle text prompts, offering greater flexibility and adaptability in real-world clinical environments. Unlike traditional models that rely on bounding boxes (Bboxes) or point prompts, this method should allow clinicians to use free-form natural language commands and streamline the diagnostic process by enabling intuitive, verbal interactions. For free-form text, we provide two conceptual definitions as follows:
(1) Anatomy-Informed Segmentation, where the user has explicit knowledge of the organ or relevant pathology to be segmented; (2) Anatomy-Agnostic Segmentation, where the user lacks medical knowledge about a specific organ or CT scan and hence queries based on positional information, organ sizes or other visible characteristics. This scenario is more common for individuals such as students or patients without formal medical training. An exemplar illustration is shown below (^1^11All of the images in this paper are best view in color.):

[backgroundcolor=gray!20, linecolor=gray!20, innertopmargin=5pt, innerbottommargin=5pt, innerleftmargin=10pt, innerrightmargin=10pt]

Example (1) : Anatomy-Informed Segmentation: An abdominal CT scan is recommended to evaluate for the presence of gallstones or any fluid accumulation around the gallbladder. — This professional diagnosis snippet indicates the most cared segmentation area is: Gallbladder.

Example (2): Anatomy-Agnostic Segmentation: I would love to have the leftmost organ segmented in this CT scan. — This description is agnostic to the medical name of the organ, but it indicates the segmentation target from positional semantics: Liver.

To learn a free-form text-supportive MIS model, text prompt generation towards the groundtruth mask is a primary step. Instead of using labor-intensive manual labeling to match with the masks, we propose a retrieval augmented generation (RAG) fashion [] method that automates text query generation using corpus embeddings collected from three resources (clinical expert records, non-expert queries, and synthetic queries). This approach guarantees that the generated query prompts capture various forms of language use across different demographic groups. Based on the text queries, we propose FLanS, a free-form language-based segmentation model that can accurately interpret and respond to free-form prompts either professional or straightforward, ensuring accurate segmentation across a variety of query scenarios.

Another challenge in text-based medical imaging segmentation arises from the variability in scan orientation. Factors such as patient positioning (e.g., supine vs. prone), different imaging planes (axial, coronal, sagittal), reconstruction algorithms and settings, and the use of portable imaging devices in emergency settings can cause organs to appear in unexpected locations or orientations. The scan orientations even differ between well-preprocessed datasets, such as AbdomenCT-1K [] and BTCV [], as shown in Fig. 1. This variability can confuse segmentation models, making it difficult to distinguish between the anatomical position of an organ and its appearance in a scan. For instance, the right kidney may appear either on the left or the right side of a rotated scan, leading to inaccurate segmentations. To address this challenge, we integrate the symmetry-aware canonicalization module as a crucial step in our model architecture [, ], which ensures the model produces consistent segmentations regardless of the scan’s orientation, enhancing its accuracy across diverse medical images [, ].
Additionally, incorporating symmetry improves sample efficiency and generalizability, which is well-suited for medical imaging tasks where labeled datasets are limited [, , , ].

Our key contributions in this paper are summarized as follows:

- •
We employ RAG techniques for free-form text prompt generation for various anatomical structures containing diverse anatomy-informed and anatomy-agnostic queries. Stems from the vectorized embedding of clinical reports, produced query data employs the realistic tones and word usage.
- •
We present a novel medical image segmentation model, FLanS, that exhibits a deep understanding of the relationship between text descriptions and medical images. It uniquely supports free-form text segmentation and employs a symmetry-aware canonicalization module to handle variability in scan orientation, as in Table. 1.
- •
Our model training uses $\sim$100k medical images from 7 public datasets, covering 24 organs, along with diverse text prompts. This ensures the model generalizes across diverse anatomical structures and clinical scenarios and can be easily extended to new organs with upcoming datasets.
- •
We demonstrate the FLanS’s effectiveness on both in-domain and out-of-domain datasets, and perform ablation studies to validate the contributions of each component in our model design.

Figure: Table 1: FLanS uniquely supports all prompt types, including free-form text, and is symmetry-aware.
Refer to caption: x2.png

## 2 Related Work

#### Medical Image Segmentation

Medical image segmentation (MIS) aims at accurately delineating anatomical structures in medical images. Traditionally, MIS methods tend to segment the correct regions from an image that accurately reflects the input query []. The researchers improve the performance of MIS methods by either optimizing segmentation network design for improving feature representations [, , , ], or improving optimization strategies, e.g., proposing better loss functions to address class imbalance or refining uncertain pixels from high-frequency regions to improve the segmentation quality [, , ]. However, they require a pre-known medical region from the user as an input for segmentation on where it is expected to be segmented and a precise match between the segment’s name and the labels used in the training set, restricting their flexibility in real-world application. Another category of methods are SAM-based approaches [, , ] that mainly rely on the Bboxes or points as prompts for segmentation. While such methods do not need strict labels, they neglect the descriptive understanding of the image, revealing a deficiency in performing arbitrary description-based segmentation, in comparison, our method handles well in Labels, free-form Text prompts without losing ability of Point and Bbox, as shown in the Table. 1.

#### Text Prompt Segmentation

Text prompt segmentation, also referred to as expression segmentation [], utilizes natural language expressions as input prompts for image segmentation tasks, moving beyond the traditional reliance on class label annotations []. Early research in this area employed CNNs and RNNs for visual and textual feature extraction, which were later combined through feature fusion for segmentation []. The success of attention mechanisms further inspired a new line of work [, ]. More recently, transformer-based architectures have improved segmentation performance by using either carefully designed encoder-based feature fusion modules [, , ] or decoder-based approaches [, , ]. Among these, [] introduced a text-promptable mask decoder for efficient surgical instrument segmentation. However, there is no existing work that has focused on free-form language segmentation for diagnosis-related medical imaging tasks as introduced in this work.

#### Equivariant Medical Imaging

Equivariant neural networks ensure that their features maintain specific transformation characteristics when the input undergoes transformations, and they have achieved significant success in various image processing tasks [, , , ]. Recently, equivariant networks have also been applied to medical imaging tasks, including classification [], segmentation [, , ], reconstruction [], and registration []. Equivariance can be incorporated in different ways, such as through parameter sharing [], canonicalization [], and frame averaging []. In our work, since we leverage a pretrained segmentation network, we achieve equivariance/invariance through canonicalization [], which, unlike other methods, does not impose architectural constraints on the prediction network.
It uses a simple equivariant canonicalization network that transforms the input to a canonical form before feeding it to an unconstrained prediction network. By leveraging this technique, the performance and robustness of our model are greatly enhanced.

## 3 Methodology

In this section, we introduce a paradigm to equip the segmentation model with free-form language understanding ability while maintaining high segmentation accuracy. It employs the
RAG framework to generate text prompts based on real world clinical diagnosis records. The generated free-form queries, anchored on the corresponding organ labels, are used to train a text encoder capable of efficiently interpreting the segmentation intentions (e.g., different interested organs disclosed in anatomy-informed or anatomy-agnostic prompts) and guiding the segmentation network. We also incorporate a canonicalization module, which can transform input images with arbitrary orientations into a learned canonical frame, allowing the model to produce consistent predictions regardless of the input image orientation.

#### Preliminaries of SAM Architecture

SAM [] contains three main parts: (1) an image encoder that transforms images into image embeddings; (2) a prompt encoder that generates prompt embeddings; (3) a mask decoder
that outputs the expected segmentation mask based on the image and prompt embeddings. Given a corresponding input medical image $x\in\mathcal{X}$ and a relevant prompt $p\in\mathcal{P}_{x}$. The image encoder embeds $x$ into $z_{x}$ that $z_{x}=\text{Encoder}^{\mathcal{X}}(x)$, similarly the prompt embedding $z_{p}=\text{Encoder}^{\mathcal{P}}(p)$. The mask decoder predicts the segmentation result (mask) by $\hat{m}_{x}^{p}=\text{Decoder}(z_{x},z_{p})$. While the SAM model provides $\text{Encoder}^{\mathcal{P}}$ for spatial prompts (e.g. Bbox or point), the integration of text-based prompts has been less explored. In text-based medical images segmentation, natural language prompts require specialized learning to effectively capture clinical terminology and segmentation intent.

### 3.1 The Retrieval Augmented Query Generator

#### Anatomy-Informed Query

To equip a MIS model $\mathcal{M}$ with language comprehension abilities, it is essential to prepare a suitable natural language query (^2^22Throughout the paper, we use the terms “query” and “prompt” interchangeably.) corpus $\mathcal{C}$ in correspondence with the target

Figure: Figure 2: The RAG Free-form Query Generator. The domain corpus, from the EMRs embedding, completes the retrieval augmentation and enhances the LLMs with the clinical way of query.
Refer to caption: x3.png

organ label set $\mathcal{L}=\{l_{1},l_{2},...l_{n}\}$, where $l_{1}=\textit{Liver}$, $l_{2}=\textit{Kidney}$, etc., as in Appendix Fig. 9. Since manual annotation is time-consuming and can be biased towards individual linguistic habits, we designed a RAG-based free-form text prompts generator to automate this process. RAG allows pre-trained LLMs to retain their free-form language generation capabilities while incorporating domain-specific knowledge and style from the provided data source $\mathcal{S}$. We collect corpus from three types of data sources. Two of these, $\mathcal{S}_{1}=\textit{Domain Expert}$, $\mathcal{S}_{2}=\textit{Non-Expert}$, serve as the corpus set to simulate various styles of descriptions for segmentation purposes,. The third source, $\mathcal{S}_{3}=\textit{Synthetic}$, is directly generated by GPT-4o to imitate descriptions for segmentation purposes.

For $\mathcal{S}_{1}=\textit{Domain Expert}$, we collected over 7,000 reports written by doctors and identified 4,990 clinical diagnosis records that are relevant to 24 labeled organs for this study. After de-identification, we embed such Electronic Medical Records (EMRs) into semantic vector space through Med-BERT [], which outperforms the general language embedding models such as Bert or GPTs in the bioinformatics context understandings.
Then, we built a retrieval augmented generation fashion generator agent $\mathbf{G}$, as shown in Fig. 2, provided with medical domain corpus and practitioner’s language usage preference. It retains the original LLM’s natural language ability such as sentences extension and rephrasing. Finally, we construct a query prompt template: “System: You are an agent able to query for segmenting label {Liver} in this {CT} scan. Please write the query sentence and output it.” Given a label $l_{i}$ = Liver, where $l\in\mathcal{L}$ regarding an arbitrary organ label with CT modality, the $\mathbf{G}$ produces a free-form query $q_{l}^{i}$, this query is taken as prompt in the later text-aware segment model training. E.g., “(1) Examine this CT scan to determine the extent of hepatic damage present. (2) As the symptoms suggest cirrhosis, we should analyze the related part in this CT scan for any signs of the disease”. These retrieved augmented results show that the interested organ may not always be explicitly mentioned, but can be inferred based on terms like ‘cirrhosis’ and ‘hepatic’, which are all liver-specific illnesses in clinical practice.

For $\mathcal{S}_{2}=\textit{Non-Expert}$, we collected queries from people without medical training who lack knowledge of the anatomy structures to formulate the segmentation queries. For $\mathcal{S}_{3}=\textit{Synthetic}$, the corpus is directly generated by LLMs. Both $\mathcal{S}_{2}$ and $\mathcal{S}_{3}$ are combined with $\mathcal{S}_{1}$ and processed by $\mathbf{G}$ to produce diverse and rich expression text queries for any given organ.

#### Anatomy-Agnostic Query

Anatomy-agnostic queries are crucial for training models to handle more plain descriptions (i.e., positions, sizes) that lack explicit organ names or related anatomy information. To align the anatomy-agnostic queries, $\mathcal{Q}$, with training images and their ground truth masks, we follow the process shown in Fig. 3. Given a training sample $x$, we first retrieve spatial information for each of its mask $m_{x}^{(i)}$ using Bboxes, deriving spatial categories based on their positions and sizes, $k\in{\mathcal{K}}$, where the set $\mathcal{K}=\{k^{1*},k^{2*},\dots,k^{6*}\}$ represents six categories: largest, smallest, left-most, right-most, upmost, and bottom. The RAG generator $\mathbf{G}$ then extends this information into full language descriptions for the masks that belong to one of these six categories, generating anatomy-agnostic text queries to augment $\mathcal{P}_{x}$ for each $x\in\mathcal{X}$.
This pipeline, as Fig. 3, ensures sufficient anatomy-agnostic queries are provided to train the model to segment the accurate organ masks without needing to know the organ label names.

Figure: Figure 3: Spatial features extracted from the Bboxes of ground truth masks are processed by the RAG query generator $\mathbf{G}$ to produce anatomy-agnostic queries.
Refer to caption: x4.png

### 3.2 Free-Form Language Segmentation for Medical Images

After generating a large corpus of free-form text queries via our retrieval augmented query generator, the next step is to align these queries with medical imaging segmentaion tasks.

#### Anatomy-Informed Segmentation

For free-form anatomy-informed text prompts, the text encoder must learn embeddings that group similar organ segmentation intents together while clearly separating unrelated intents in distinct semantic clusters.
We adopt the CLIP [] as the foundation of text encoder for its capability of understanding semantics. Given a text prompt $p\in\mathcal{P}_{x}$ associated with the image $x$,
the CLIP text encoder converts it into an embedding vector $\mathbf{t}_{p}$ in a shared embedding space: $\mathbf{t}_{p}=\text{Encoder}^{\mathcal{P}}(p)\in\mathbb{R}^{D}$,
where $D$ is the dimensionality of the text embedding space.
To further strengthen the model’s ability to differentiate between organ segmentation, we introduce an intention head on top of the text embeddings by CLIP. This head is a linear layer $\mathbf{W}_{\text{cls}}\in\mathbb{R}^{C\times D}$, where $C=24$ is the number of organ class. The intention logits $\mathbf{y}_{p}$ are derived for each encoded vector $\mathbf{t}_{p}$: $\mathbf{y}_{p}=\mathbf{W}_{\text{cls}}\mathbf{t}_{p}+\mathbf{b}_{\text{cls}}$.
Given a corresponding medical image embedding $z^{x}$, we train the model by following loss function:

$$ $L=\operatorname*{arg\,min}_{\{\mathbf{W}_{\text{cls}},\mathbf{b}_{\text{cls}}, \mathbf{W}^{E},\mathbf{W}^{D},\mathbf{W}^{P}\}}\frac{1}{|\mathcal{X}|}\sum_{x \in\mathcal{X}}\frac{1}{|\mathcal{P}_{x}|}\sum_{p\in\mathcal{P}_{x}}[\mathcal{ L}_{\text{Dice}}(\hat{m}_{x}^{p},m_{x}^{p})+\mathcal{L}_{\text{ce}}(\hat{m}_{x }^{p},m_{x}^{p})+\mathcal{L}_{\text{ce}}(\mathbf{y}_{p},l_{p})]$ (1) $$

where $\hat{m}_{x}^{p}=\text{Decoder}(z_{x},\mathbf{t}_{p})$ and $m_{x}^{p}$ are predicted and ground truth masks. $l_{p}\in[0,...,23]$ is the ground truth organ class for the prompt.
$\mathbf{W}^{E}$, $\mathbf{W}^{D}$ and $\mathbf{W}^{P}$ represent the image encoder, decoder and CLIP text encoder weights, respectively. We use both Dice loss $\mathcal{L}_{\text{Dice}}$ and cross-entropy loss $\mathcal{L}_{\text{ce}}$ for predicted masks. The classification loss $\mathcal{L}_{\text{ce}}(\mathbf{y}_{p},l_{p})$ encourages the model to correctly classify organs based on text prompts, ensuring the text embedding aligns with the intended organ class.

#### Anatomy-Agnostic Segmentation

For anatomy-agnostic descriptions, which do not explicitly mention specific organs but instead focus on spatial attributes (e.g., “leftmost”, “largest”), the model must learn from spatial features $k_{x}\in\mathcal{K}$ to pair with the corresponding mask $m_{x}^{k}$ for every $x\in\mathcal{X}$. Anatomy-agnostic queries share the same embedding space as anatomy-informed queries, but $k_{x}$ is not necessarily associated with a specific organ. In this case, we use the same loss function as shown in Eq. 1 but without the last classification term.

### 3.3 Semantics-aware Canonicalization Learning

Figure: Figure 4: The architecture of our proposed model FLanS. First, given a set of free-form text prompts $p_{1}...,p_{n}$, the text encoder gets the text embedding, and then passes through the learned Intention Head Layer that maps the embedding to a space with explicit intention probabilities, which is useful for the FLanS model weight updating as in Eq. 1. Second, we have trained a Canonicalized Network that transforms any medical image with arbitrary orientation into a canonicalization space, making sure the encoded image aligns with the standard clinical practice to avoid ambiguity. Third, the encoded prompts (either spatial info such as Point, Bbox, or Free-form text data), together with the encoded image, will be processed with mask decoder and output the expected masks.
Refer to caption: x5.png

We incorporate roto-reflection symmetry [] into our architecture for two key reasons: 1) Organs and anatomical structures can appear in various orientations and positions due to differences in patient positioning, imaging techniques, or inherent anatomical variations. Equivariance ensures that the model’s segmentation adapts predictably to transformations of the input image. 2) We aim to ensure our model reliably interprets and segments organs that have positional terms in their names, such as “left” or “right kidney” from text prompts regardless of the scan’s orientation, thereby enhancing the model’s robustness and accuracy.

Following [, ], we train a separate canonicalization network ${h}:\mathcal{X}\mapsto G$, where $\mathcal{X}$ represents the medical image sample space, $G$ represents the desired group, and $h$ is equivariant to $G$. This network generates group elements that transform input images into canonical frames, standardizing the image orientation before applying the prediction function. The Eq. 2 shows how this canonicalization process maps the transformed input back to a common space where the segmentation prediction network $p$ operates,

$$ $f(x)=\rho_{\text{out}}({h}(x))\;p(\rho_{\text{in}}({h}^{-1}(x))x,\;\textbf{t})$ (2) $$

Where $p$ is the segmentation prediction network (composed of the Image Encoder and Mask Decoder in Figure 11), t is the text prompt embedding produced by our text encoder, and $\rho_{\text{in}}$ and $\rho_{\text{out}}$ are input and output representations. The segmented images or masks produced by $p$ can be transformed back with $\rho_{\text{out}}({h}(x))$ as needed. Without this transformation, $f$ is invariant; otherwise, it is equivariant. Thus, the FLanS architecture visualized in Figure 11 is invariant. We use ESCNN [] to build the canonicalization network. This approach has the advantage of removing the constraint from the main prediction network and placing it on the network that learns the canonicalization function. Appendix A provides a detailed introduction of symmetry and equivariant networks.

As the entire architecture achieves invariance or equivariance through canonicalization, the model produces the same segmentation or consistently transforms the segmentation according to the transformed input. In other words, the model always segment the same areas of interest regardless of the image’s orientation with the same text prompt. For example, as long as the ground truth “right kidney” mask of a CT image has been shown to the model once, no matter how the orientation of the CT image and the location of the right kidney changes, the model will always segment the same area.

However, without proper training, ${h}(x)$ might map different images to inconsistent canonical frames, causing a distribution shift in the inputs to the prediction network and affecting performance. Thus, training the canonicalization network togther with the segmentation prediction network is essential to ensure consistent mapping to the desired frame. It is worth noting that users can choose to disable the canonicalizer when working with anatomy-agnostic prompts, as the segmented organ may differ if the original image is not in the canonical frame. The decision depends on whether the user wants to segment the original or the canonicalized image, as the model will segment whatever image is fed into the image encoder based on the provided text prompts.

### 3.4 Training Strategy

We employ a three-stage training strategy for FLanS: 1) Learning canonicalization: we train the canonicalization network independently using FLARE22 training samples applied with random transformations from the O(2) group. The network is optimized using MSE loss between the canonicalized samples and their original counterparts. This encourages the canonicalization network to map transformed samples back to their canonical orientations as seen in the FLARE22 dataset, preventing it from selecting arbitrary orientations that could degrade the performance of the prediction network. 2) Learning text-prompted segmentation: we train FLanS with the queries from Generator $\mathbf{G}$ as introduced in Section 3.1, without the canonicalization network on the original scans, using both anatomy-informed and anatomy-agnostic prompts. This ensures that the segmentation network learns to respond accurately to different types of prompts without interference from canonicalization and data augmentation. 3) Learning augmentation and alignment: In the final stage, we perform joint training on all scans, applied with random O(2) transformations. Since the canonicalization network may not always generate the exact canonical orientation the segmentation network is accustomed to in the beginning, this serves as a form of free augmentation for the segmentation networks. Over time, the canonicalization and segmentation networks align.

## 4 Experiment

### 4.1 Datasets and Experiments Setup

#### Image Datasets

To develop an effective organ segmentation model, we collected 1,437 CT scans from 7 public datasets, covering 24 partially labeled organs. Of these, 1,089 scans from MSD [], BTCV [], WORD [], AbdomenCT-1K [], FLARE22 [], and CHAOS [] are used for training. The rest 65 scans, consisting of 10% of the FLARE22 dataset (in-domain), the official validation set of WORD (in-domain), and the official test set of RAOS [] (out-of-domain), were used to evaluate model performance. To standardize the quality and reduce domain gaps across datasets, we applied pre-processing techniques such as slice filtering and intensity scaling to all CT scans. The finalized dataset comprised 91,344 images for training and validation, and 9,873 for testing. Detailed information on the dataset statistics and pre-processing steps are in Appendix B.

#### Text Datasets

Our text dataset was constructed using two types of queries: anatomy-agnostic and anatomy-informed. First, for each image, we identified organs corresponding to 6 representative positions: leftmost, rightmost, topmost, bottom, smallest, and largest. For each of these 6 position indicators, 100 anatomy-agnostic queries were generated, resulting in a set of 600 queries to serve as anatomy-agnostic segmentation prompts.(^3^33To ensure accurate position-to-organ mapping, position-driven organ-agnostic queries were applied only to images containing more than nine labeled organs during training.) Second, for each organ, we generated 480 anatomy-informed queries in an expertise-driven style using the RAG query generator. By combining both anatomy-agnostic and anatomy-informed queries, we formed a text dataset comprising 12,120 unique queries for model training. During testing, a comprehensive text set was used, containing both in-domain and out-of-domain queries. Specifically, we generated 30 RAG-generated expertise-style queries (25%, in-domain), 30 human-generated non-expertise-style queries (25%, out-of-domain), and 60 RAG-generated non-expertise-style queries (50%, out-of-domain) for each organ, forming a test set of 120 queries per organ and 2,880 queries across all organs. Detailed information on the generation of the text queries is in Appendix B.

#### Experiment Setup

All experiments were conducted on an AWS ml.p3dn.24xlarge instance equipped with 8 V100 GPUs, each with 32 GB of memory. We used a batch size of 16 and applied the CosineAnnealingLR learning rate scheduler, initializing the learning rate for all modules at 0.0001. The AdamW optimizer was employed for training. A small $D_{8}$ equivariant canonicalization network was used, consisting of 3 layers, a hidden dimension of 8, and a kernel size of 9. To maintain consistency across the input and output formats, all scans from different datasets were resized to 1024$\times$1024 and both predicted and ground truth masks were resized to 256$\times$256 for fair comparison. For images with a single channel, the channel was duplicated to 3. All models’ performance on the test sets is reported using both the Dice coefficient [] and normalized surface distance [] .

Figure: Figure 5: Left: Segmentation with anatomy-informed prompts. We could observe that FLanS can precisely segment the organ described in free-form text prompts, while other baselines make mistakes in identifying the organs. Right: Segmentation with anatomy-agnostic prompts. We could observe that the FLanS is texture-aware, descriptions of the sizes and positions can be understood, and is competitively accurate to the direct Bbox segment.
Refer to caption: extracted/5896495/fig/FLanS_Informed_Agnostic3.png

### 4.2 Anatomy-informed Segmentation

**Table 2: Anatomy-Informed Segmentation Results: FLanS consistently outperforms baselines on both organ name and free-form text prompts segmentation tasks, demonstrating superior language understanding and segmentation accuracy across in-domain and out-of-domain datasets, even when applied with random transformations.**
| Organ Name | FLARE | WORD | RAOS | TransFLARE | TransWORD | TransRAOS |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Dice | NSD | Dice | NSD | Dice | NSD | Dice | NSD | Dice | NSD | Dice | NSD |
| CLIP+MedSAM | 0.473 | 0.518 | 0.411 | 0.446 | 0.475 | 0.440 | 0.388 | 0.417 | 0.357 | 0.437 | 0.352 | 0.399 |
| MedCLIP+MedSAM | 0.557 | 0.516 | 0.466 | 0.510 | 0.419 | 0.320 | 0.485 | 0.415 | 0.342 | 0.378 | 0.336 | 0.336 |
| Universal Model | 0.649 | 0.697 | 0.512 | 0.408 | 0.442 | 0.301 | 0.380 | 0.290 | 0.299 | 0.278 | 0.200 | 0.201 |
| FLanS | 0.908 | 0.956 | 0.837 | 0.884 | 0.852 | 0.883 | 0.898 | 0.949 | 0.835 | 0.875 | 0.847 | 0.879 |

We first compare our model, FLanS, with the SOTA baselines on a held-out subset of the FLARE22 training set (FLARE), the public WORD validation set (WORD), and RAOs cancer CT images (RAOS). Both FLARE22 and WORD serve as in-domain test sets, while RAOS is an out-of-domain test set, as neither our model nor the baselines were trained on this dataset. Although the original test sets already contain scans with varying orientations, we further evaluated the models’ robustness by applying random O(2) transformations to the three test sets, creating additional sets: TransFLARE, TransWORD, and TransRAOS. More importantly, we tested the models using Anatomy-Informed text prompts, which included two types: purely organ names and free-form text descriptions.

As for the baselines, the Universal Model [] is the only published medical imaging foundation model that considers free-form text descriptions. This model integrates text description embeddings during training, while segmentation at the testing and inference stages is performed using organ IDs. Consequently, we evaluate this model with prompts consisting solely of organ names. Another widely used approach for text-prompt segmentation involves combining CLIP-based models [] with segmentation models [, ]. In these methods, segmentation models first generate potential masks based on a set of random bounding box or point prompts that span the entire image. CLIP-based models then embed both the text prompt and the cropped images from these masks. The final mask is selected based on the highest similarity between the cropped image embedding and the text embedding. To cover this approach, we include two additional baselines: 1) CLIP + MedSAM, where MedSAM [] is SAM [] fine-tuned on medical imaging datasets; and 2) MedCLIP + MedSAM, where MedCLIP [], a contrastive learning framework trained on diverse medical image-text datasets, is paired with MedSAM for segmentation.

As we can see from Table 2, FLanS achieves superior performance in segmenting based on organ name. More importantly, FLanS significantly outperforms the baselines on free-form text prompts segmentation, where the baselines struggle with more complex language input. This suggests that training with diverse free-form text prompts enhances the model’s ability to understand language and the relationship between text descriptions and medical images. Furthermore, FLanS maintains high Dice and NSD scores on the transformed test sets thanks to the help of the canonicalization network. The left panel of Fig. 5 visualizes the segmentations generated by the best baseline and FLanS, alongside their corresponding text prompts, illustrating our model’s superior language understanding and segmentation accuracy.

### 4.3 Anatomy-Agnostic Segmentation

**Table 3: Anatomy-Agnostic Segmentation Results: Comparison of FLanS using positional and size information text prompts vs. MedSAM and SAM2 using Bboxes or points. FLanS achieves competitive or superior performance across both in-domain and out-of-domain test sets.**
| Methods | FLARE | WORD | RAOS (OOD) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Dice | NSD | Dice | NSD | Dice | NSD |
| SAM2-large (Point-prompt) | 0.585 | 0.652 | 0.534 | 0.551 | 0.488 | 0.497 |
| SAM2-large (Bbox-prompt) | 0.873 | 0.906 | 0.848 | 0.802 | 0.818 | 0.749 |
| MedSAM (Bbox-prompt) | 0.887 | 0.872 | 0.783 | 0.781 | 0.697 | 0.681 |
| FLanS (Free-form text) | 0.844 | 0.841 | 0.855 | 0.853 | 0.851 | 0.850 |

To evaluate our model’s ability to understand anatomy-agnostic text prompts, we tested its segmentation performance using prompts that contain only positional or size-related information. To the best of our knowledge, no existing model is designed to handle anatomy-agnostic text prompts. Therefore, we chose state-of-the-art MedSAM [] (SAM fine-tuned on medical imaging datasets) and the latest SAM2 [] as baselines. However, instead of text prompts, these models were provided with ground-truth organ Bboxes or point prompts. Our goal in this experiment is for FLanS to achieve comparable results to the baselines because FLanS is only given text prompts with positional or size information while the baselines are given the bounding box or point prompts of ground truth organ.

As shown in Table 3, FLanS the best or second-best performance across both in-domain and out-of-domain test sets. MedSAM performs well on the FLARE and WORD test sets but struggles on the RAOS test set due to the lack of training on that dataset. SAM2, when provided with bounding box prompts, consistently performs well across all test sets and demonstrates strong generalizability. However, its performance significantly degrades with point prompts, likely because medical scans lack the distinct edges present in the datasets SAM2 was originally trained on. The right panel of Fig. 5 visualizes the segmentations produced by the best baseline and FLanS, along with their corresponding anatomy-agnostic text prompts. It demonstrates that FLanS can reliably segment the correct organs based on the provided positional or size information, such as largest and lower right.

### 4.4 Ablation Study on the Model Architecture

Figure: Figure 6: The model without canonicalization incorrectly highlights the left kidney due to confusion between anatomical position (“right kidney”) and the organ’s appearance on the right side of the image.
Refer to caption: extracted/5896495/fig/can_vs_baseline.png

We conducted an ablation study of FLanS on the FLARE22 dataset [] to understand the contribution of each component, as presented in Table 4. Using an 80%-10%-10% train-validation-test split on the public FLARE22 training set, we evaluate the models’ performance on both the held-out test set and a transformed test set, which contained samples applied with random transformations from $O(2)$. Table 4 shows the prediction performance of FLanS and its variants, with components progressively removed. The results highlight that each component plays a crucial role in the model’s overall performance. Notably, while data augmentation improved the model’s robustness to random transformations, it slightly reduced performance on the canonical test set, as the model had to handle various transformations. However, by canonicalization network, the segmentation backbone focuses specifically on canonicalized medical images, thus achieving the best performance on both test sets.

### 4.5 Effective Understanding of Free-Form Text Prompts

Fig. 7 left visualizes the t-SNE embeddings of free-form text prompts corresponding to all 13 FLARE22 data classes, including liver, right kidney, spleen, and others. The text prompt encoder effectively clusters these prompts, revealing anatomically structured semantics. This demonstrates FLanS has a strong capability in understanding and distinguishing free-form text prompts.

Figure: Figure 7: Left: t-SNE visualization of the free-form text prompt embedding space. Our method can effectively distiguish between different organ related queries. Right: Canonicalized CT scans from $D_{4}$ and $D_{8}$ canonicalization networks for a batch of randomly transformed scans from the FLARE22 dataset. Medical images can be successfully transformed back to an aligned canonicalization space.
Refer to caption: extracted/5896495/fig/tsne_visualization_5.png

### 4.6 Effectiveness of the Canonicalization

The right side of Fig. 7 shows the canonicalized CT scans from $D_{4}$ and $D_{8}$ canonicalization networks for a batch of original scans from the FLARE22 dataset applied with random transformations from $O(2)$ group. As the group order of the canonicalization network increases, the scans become more consistently aligned to a particular canonical orientation. The canonicalization networks use a shallow architecture with three layers, a hidden dimension of 8, and a kernel size of 9, demonstrating that even a simple network with a larger kernel can effectively achieve canonicalization.

More importantly, applying canonicalization before feeding the scans into the main segmentation network and making the entire architecture equivariant or invarianthelps prevent confusion caused by positional terms in the organ name. A text-prompt segmentation model understands positional cues such as “left” vs “right” but it may get confused between the anatomical position and the organ’s appearance in the scan. For example, Fig. 6 shows segmentation predictions from models with and without canonicalization, given the anatomy-informed text prompt, “Highlight the right renal organ.” Since the CT scan is not in the standard orientation, the right kidney appears on the left side of the image. Without canonicalization, the non-equivariant model incorrectly segments the left kidney, which appears on the right side. Our model can make consistent predictions of the right kidney regardless of the scan’s orientation, allowing it to focus on learning the critical features of the organs.

## 5 Conclusion

In this work, we presented FLanS, a novel medical image segmentation model capable of handling diverse free-form text prompts, including both anatomy-informed and anatomy-agnostic descriptions. By integrating equivariance, our model ensures accurate and consistent segmentation across varying scan orientations, addressing a critical challenge in medical imaging. We also developed a RAG query generator for both realistic and synthetic prompt generation, and trained FLanS on over 100k medical images from 7 public datasets, covering 24 organ categories. FLanS outperforms baselines in both in-domain and out-of-domain tests, demonstrating superior language understanding and segmentation accuracy. Future works including extend FLanS to multi-organ segmentation tasks and further enhance RAG generator with multimodal data.

## Appendix A Equivariance and Symmetry

Equivariant neural networks are designed to explicitly incorporate symmetries that are present in the underlying data. Symmetries, often derived from first principles or domain knowledge, such as rotational or translational invariance, allow the network to process inputs in a way that is consistent with these transformations. This is particularly important when the ground truth functions respect such symmetries, as the incorporation of these properties can significantly enhance model performance and generalization.

#### Group

A group of symmetries or simply group is a set $G$ together with a binary operation $\circ\colon G\times G\to G$ called composition satisfying three properties: 1) identity: There is an element $1\in G$ such that $1\circ g=g\circ 1=g$ for all $g\in G$; 2) associativity: $(g_{1}\circ g_{2})\circ g_{3}=g_{1}\circ(g_{2}\circ g_{3})$ for all $g_{1},g_{2},g_{3}\in G$; 3) inverses if $g\in G$, then there is an element $g^{-1}\in G$ such that $g\circ g^{-1}=g^{-1}\circ g=1$.

Examples of groups include the dihedral groups $D_{4}$ (symmetries of a square) and $D_{8}$ (symmetries of an octagon), as well as the orthogonal group $O(2)$, which represents all rotations and reflections in 2D space. Both $D_{4}$ and $D_{8}$ are discrete subgroups of $O(2)$.

#### Representation

A group representation defines how a group action transforms elements of a vector space by mapping group elements to linear transformations on that space. More specifically, a group representation of a group $G$ on a vector space $V$ is is a homomorphism: $\rho\colon G\to\text{GL}(X)$, where $\text{GL}(X)$ is the group of invertible linear transformations on $V$. This means for any $g_{1},g_{2}\in G$, $\rho$ is a linear transformation (often represented by a matrix) such that the group operation in $G$ is preserved:

$$ $\rho(g_{1}g_{2})=\rho(g_{1})\rho(g_{2})$ (3) $$

#### Equivariance

Formally, a neural network is said to be equivariant to a group of transformations $G$ if applying a transformation from the group to the input results in a corresponding transformation to the output. Mathematically, for a function $f\colon X\to Y$ to be $G$-equivariant, the following condition must hold:

$$ $f(\rho_{\text{in}}(g)(x))=\rho_{\text{out}}(g)f(x)$ (4) $$

for all $x\in X$ and $g\in G$, where $\rho_{\text{in}}\colon G\to\text{GL}(X)$ and $\rho_{\text{out}}\colon G\to\text{GL}(Y)$ are input and output representations []. Invariance is a special case of equivariance where the output does not change under the group action. This occurs when the output representation $\rho_{\text{out}}(g)$ is trivial. Figure 8 visualize how the equivariant and invariant networks work.

Figure: Figure 8: An equivariant model (left) ensures that its output transforms in a specific, predictable way under a group of transformations applied to the input, preserving the structure of the transformation (e.g., rotating the input results in a correspondingly rotated output). In contrast, an invariant model (right) produces an output that remains unchanged regardless of any transformations applied to the input from the same group.
Refer to caption: extracted/5896495/fig/symmetry_definition.png

#### Equivariance via weight-sharing

One of the primary approaches to incorporating symmetry into neural networks is through weight sharing [, , ]. This approach enforces equivariance by constraining the network’s architecture so that the weights are shared across different group elements. For example, in $G$-convolutions [], the same set of weights is shared across the transformed versions of the input, ensuring that the network’s predictions remain consistent under those transformations. In a layer of $G$-steerable CNNs [], a set of equivariant kernel bases is precomputed based on the input and output representations, and the convolution kernel used is a linear combination of this equivariant kernel basis set, where the coefficients are trainable. Similar approaches can also be used to develop equivariant graph neural networks []. These architectures directly modify the network’s layers to be equivariant, ensuring that each layer processes symmetries in a way that is aligned with the desired group. While powerful, this approach imposes architectural constraints, which may limit the flexibility of the network and prevent leveraging large pretrained models.

#### Equivariance via canonicalization

An alternative to weight sharing is incorporating symmetry through canonicalization [, ], where, instead of modifying the network’s architecture to handle symmetries, the input data is transformed into a canonical form. In this approach, a separate canonicalization network, which is itself equivariant, preprocesses the input, transforming it into a standard, or canonical, representation. This canonicalized input is then passed to a standard prediction network that does not need to be aware of the symmetries. If the corresponding inverse transformation is applied to the output of the prediction network, the entire model becomes equivariant; otherwise, the model remains invariant. This method has several advantages. First, it does not require altering the architecture of the prediction network, allowing for the use of large pre-trained models without modification. Second, by ensuring that the input data is in a canonical form, the prediction network only needs to learn the mapping from the canonical input to the output, without needing to learn all transformed samples. This can lead to improved performance and robustness, especially in scenarios where the prediction task does not naturally align with the symmetry group or where architectural constraints might hinder performance. Thus, in our work, we leverage canonicalization to achieve equivariance in the segmentation task. By transforming the input into a canonical form using a simple equivariant canonicalization network, we ensure that our prediction network remains unconstrained and can fully utilize its capacity for learning without the need for architectural modifications. This approach offers the benefits of symmetry-aware processing while maintaining the flexibility and power of unconstrained neural network architectures.

## Appendix B Detailed Dataset Description

#### Image Data Collection and Preprocessing

For model development and evaluation, we collected 1,437 CT scans from 7 public datasets. A detailed summary of the datasets is provided in Table 5. In total, 24 organs are labled in the assembled datasets, with a strong focus on segmentation targets in the abdominal region. The organ class distribution across the datasets is shown in Fig 9. To standardize quality and reduce domain gaps, we applied a preprocessing pipeline to all datasets. Specifically, we mapped the Hounsfield unit range [-180, 240] to [0, 1], clipping values outside this range. To address dimension mismatches between datasets, masks, and images, all scans and masks were resized to $1024\times 1024$. The 3D scan volumes were sliced along the axial plane to generate 2D images and corresponding masks. To ensure labeling quality, organ segments with fewer than 1,000 pixels in 3D volumes or fewer than 100 pixels in 2D slices were excluded. The finalized dataset consisted of 101,217 images, with 91,344 (90.25$\%$) used for training and validation, and 9,873 (9.75$\%$) reserved for testing.

**Table 5: Overview of the datasets used in this study.**
| Dataset | # Training scans | # Testing scans | Annotated organs(^1^11For simplicity, the following abbreviations are used: Liv (liver), Kid (kidney), Spl (spleen), Pan (pancreas), Col (colon), Int (intestine), Sto (stomach), LKid (left kidney), RKid (right kidney), Aor (aorta), Eso (esophagus), IVC (inferior vena cava), Duo (duodenum), RAG (right adrenal gland), LHF (left head of femur), Bla (bladder), Rec (rectum), Gal (gallbladder), LAG (left adrenal gland), RHF (right head of femur), PVSV (portal vein and splenic vein), Pro (prostate), and SV (seminal vesicles).) |
| --- | --- | --- | --- |
| AbdomenCT-1K | 722 | — | Liv, Kid, Spl, Pan |
| MSD(^2^22Only the lung and spleen subsets from MSD were used.) | 157 | — | Lun, Spl |
| WORD | 100 | 20 | Liv, Spl, LKid, RKid, Sto, Gal, Eso, Pan, Duo, Col, Int, LAG, RAG, Rec, Bla, LFH, RFH |
| FLARE22 | 40 | 5 | Liv, RKid, Spl, Pan, Aor, IVC, RAG, LAG, Gal, Eso, Sto, Duo, LKid |
| CHAOS | 40 | — | Liv |
| BTCV | 30 | — | Spl, RKid, LKid, Gal, Eso, Liv, Sto, Aor, IVC, PVSV, Pan, RAG, LAG |
| RAOS(^3^33We used CancerImages (Set1) from RAOS as our out-of-domain test set. To avoid overlap, any scans in RAOS that were extended from WORD were excluded from testing.) | — | 40 | Liv, Spl, LKid, RKid, Sto, Gal, Eso, Pan, Duo, Col, Int, LAG, RAG, Rec, Bla, LFH, RFH, Pro, SV |

#### Test Data Creation

Different from existing work that solely chases for a higher segmentation accuracy, in this paper, we expect to evaluate the segment model’s performance in dual tasks: The free-form text understanding ability and segmentation ability.

Figure: Figure 9: Distribution of labeled organs across the collected datasets. The image count for each organ and its corresponding ratio is marked in the plot.
Refer to caption: extracted/5896495/fig/data_statistics.png

In order to verify the model’s ability to understand the language descriptions, we construct a query dataset (test set) from two resources: 1. Real-world human queries; 2. LLM-generated synthetic queries. For the first kind of real-world queries, we have two groups of annotators, Domain Expert and Non-Expert. Domain experts are from clinical hospitals who provide the query materials from their daily diagnosis notes, this group of people tends to use professional vocabulary, and their intention might not be explicitly expressed in a professional report, such as in the report, the doctor writes ‘Concerns in the hepatic area
that warrant a more focused
examination’, which implicitly means the ‘liver is the area of interest under certain symptom’. Another group of query providers is the non-expert, who are not specialized in clinical or equipped with medical specialties. We explain to this group of people that their task is to write a sentence and show the intention of segmenting the target organ/tissue in a CT scan, e.g., the liver. This aspect of real queries represents a more general and non-specialist approach to expressing the need for segmentation (such as in the student learning scenarios). Apart from real query data, we incorporate synthetic test queries to enlarge the test samples and add randomness in various expressions. The synthetic test is generated by GPT-4o following the template shown below:

The Prompt Template to Generate Synthetic Queries.

System Description: You are a doctor with expert knowledge of organs.

Task Description: Now you are making a diagnosis of a patient on the CT scan over {body part}. You find a potential problem on {organ name} and want to see more details in this area, please query for segmentation by free-form text. Please make sure to deliver the segment target explicitly, and you are encouraged to propose various expressions.

Format: {segmentation query}, {explain reason}.

Example: Given that, {body part} is abdomen and {organ name} is liver.

Your response should be something like: {Please identify the liver for me for more analysis.} {Because elevated liver enzymes alanine aminotransferase (ALT) in the blood tests might indicate liver inflammation or damage}.

Output: {Placeholder}

The overall structure of the test dataset is shown in Figure 10. It consists of 25% expert queries, 25% normal queries, and half synthetic queries. In total, we have 2880 (24 organs x10 queries x3 x2x2) text queries. Each of the queries is labeled with the correct organ name to segment. This will be used to evaluate the ability of our learned TextEncoder model to understand correct intentions based on free-form language description.

At the same time, the organ names are connected to another segmentation test set, which contains several (how many) medical images such as CT scans, MRIs, etc. And stand on the results of interest-category identification, we conduct further segmentation result analysis, including the normal segmentation precision study, and also the equivariant identified segmentation study.

Figure: Figure 10: The Language Test Set for Verifying the Query Understanding Ability. It contains three aspects of components, real data - expert group, real data - non-expert group, and synthetic data.
Refer to caption: x6.png

Figure: Figure 11: Positional prompt dataset provider split, we take the slices with more than $\alpha$ labels, where we set $\alpha=8$ in this illustration (while 13 is the total label amount) as a split threshold, ensure that the slice used for training the label-agnostic provides sufficient semantics in the image content, such as left, upmost or largest, etc. Similarly, we process the other datasets such as BTCV and WORD.
Refer to caption: extracted/5896495/fig/split_position_dist.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1) When we perform training with medical images, changing the orientation to a consistent orientation (i.e. RAS) is a usual step to preprcoess the data. It will be great if you can clarify why you think it is a clinical problem and needs a machine learning model to correct the orientation?

2) From Figure 3, you are using point / id to identify the organ semantic in the image and want to use the generator to correspond each organ with text, which is an interesting idea. However, how about some small organs in the abdominal region? It is challenging to describe all the organ location is, as they are nearby and is there any experimental setting that you have tried to adapt all organs?

3) Similar work have been done by adapting text into segmentation, please cite this paper in the related work and it will be great to see if there is an experiment comparison:
- Zhao, Theodore, et al. "BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once." arXiv preprint arXiv:2405.12971 (2024).

4) For Figure 7, the t-SNE plot for text-prompt embedding should be completely separable, because the text is pointing towards different organs. A meaningful visualization towards your model should be an attention map of computing the correlation between text and the image. It will be great to have the attention map that demonstrates the shape of the corresponding organs and show the correlated semantics between text and our segmentation target.
soundness: 3
strengths: The strength of this paper can be summarized as follows:
1) It is interesting to see how the variable of text description affects the segmentation performance
2) Adapting RAG to generate description is an interesting way to tackle the generalization capability using text
summary: This paper aims to explore the capability of text for segmentation in real clinical setting, the main contributions of this paper can be summarized as following:
1) Proposed a retrieval-augmented generation (RAG) based free form text prompt generator to enhance the segmentation performance
2) Proposed a symmetry-aware canonicalization module to ensure same scan orientations and reduce anatomical position
3) Demonstrated effectiveness on both in-domain and out-of-domain datasets
weaknesses: The weakness of paper can be summarized as follows:
1) Lack of experiments of comparing with 2D current state-of-the-art supervised model (i.e. nn-UNet, Swin-UNet) 
2) The image orientation to train a model cannot be counted as a novel problem to handle, as it is not a problem and it is just a correction to the experiment setting.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Q1: What are the use cases where FLanS is preferable over a TotalSegmentator? / Why shouldn't I ask ChatGPT to predict the class label and then just have a script retrieve the TotalSegmentator mask instead? (For anatomy-informed prompts)
Q2: How did you account for the "Segment the leftmost organ" not changing when iterating along 2D slices? -- How would you imagine this prompt should be applied in practice?
Q3: Did you evaluate fully in 3D or did you only consider a 2D slice by itself?

If the authors are able to provide experiments regarding A, B or C in the rebuttal (should be do-able since it should be just about providing new prompts) or convince me that their free-form text organ segmentation shows benefits over a TotalSegmentator model I am certainly willing to raise my score.
soundness: 2
strengths: I agree with the authors that language inclusions are currently underutilized in the domain of medical image segmentation and I like their creation of additional text reports through the RAG approach. Moreover, the description of position to steer predictions is novel.
summary: ### Summary:
This paper introduces a free-form language-based segmentation algorithm termed FLanS. This model allows language-prompt-based segmentation of 24 different organs by creating text prompts in a RAG fashion. They differentiate into two type of prompts, anatomy-informed and anatomy-agnostic text prompts that are supposedly commonly used in practice and should allow for easier segmentation given a new clinical image. The authors evaluate their method on a variety of abdominal organ CT downstream datasets, namely FLARE22, WORD and RAOS showing improved performance over the chosen baselines.
weaknesses: I am highly skeptical that the current capabilities of the method are introducing new capabilities over existing methods in any meaningful way, which I will elaborate on in the following.

### Main concerns:
While I believe the integration of language to steer segmentation is very useful, I believe the author’s current use-case is not convincing at all. The following are the main points of critique:
1. Every clinician or experienced user is able to easily state which organ they are interested in. There is no added benefit of trying to predict which organ one refers to from a report description, this raises questions about the anatomy-informed prompt segmentation and also compromises the contribution of the RAG component that emulates clinical prompts.
2. The anatomy-agnostic setting is also not useable in the current format: In every clinical setting patients are present in 3D format. Given that this method is 2D, the “largest” organ will not be consistent across all 2D slices when one does whole-volume inference. So if one wants to use this method the clinician/user would have to adapt the slices to infer or create unique prompts for unique slices, which is both very unpractical.
3. Due to the proposed method being closed-set and being constrained to basic organ segmentation, any supervised model that does organ segmentation (and predicts all organs) would currently have a very similar capability, without the prior issues. The user would just have to discard all segmentations he is not interested in.
4. Regarding canonicalization, I would like to see the prevalence of these cases. Generally, all 3D images come with meta-information in the image header that should allow to re-orient it in a canonical way already. Hence I would like to have a quantification of how often this overall occurs to convince me that this is actually a problem worth solving.  
5. Evaluation: It currently seems like the authors are only testing their capabilities in a closed-set setting. The final anatomy-informed prompts are the same as they used during training. To actually show that their method provides novel capabilities experiments where they try to predict novel classes given the text guiding could be conducted, which would greatly improve the utility of their proposed method. 
6. Baselines: Since this is very close to supervised organ segmentation I would like to see supervised performance as reference (At least as a baseline to know where the performance is relative to a supervised 3D Model.

### Recommended improvements:
In my opinion, the utility of the method hinges entirely on being able to apply it in an open-set setting (a setting where they predict novel structures that they did not train on.  The way to show this would be to:
A) Predict a withheld/additional abdominal organ that was not trained on or
B) Predict a novel anatomy-agnostic prompt that they did not train on. An example for B could be: “Predict the organ above the liver.” Something that was not seen before and that could show that the model actually has learned more than the artificially created prompts.
C) Change predictive behavior for a certain organ. E.g.: “Predict the liver, but without the hepatic vessels.”
Any of these use-cases would actually bring novel capabilities, that would not be achievable or solvable by a simple supervised model. 
This would likely be much easier if the authors used pathologies and described the pathology "In the lower left lobe of the liver is a hyperintensity, please segment it for me" -- This would likely make it much easier to become open-set and do something that basic supervised methods cannot do.

Summarizing, I find the currently displayed capabilities and results not convincing, the author’s method seems to be capable of inferring which organ is desired given a prompt, which lacks overall motivation.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 3
questions: In conclusion, the paper’s primary motivation for proposing FLanS is questionable. The authors' claim of a real-world need for text-prompt-based segmentation lacks sufficient justification, given that existing tools already handle the tasks effectively. Additionally, the technical challenges outlined, such as orientation standardization, can be addressed through simpler, more established methods, making the proposed solution less compelling.
soundness: 2
strengths: 1.	The paper is well-written, and the core idea is easy to follow.
2.	The proposed method is evaluated on diverse organ segmentation datasets.
summary: The authors introduce FLanS, a medical image segmentation model capable of handling diverse free-form text prompts, including both anatomy-informed and anatomy-agnostic descriptions. By integrating equivariance, it ensures accurate and consistent segmentation across varying scan orientations. Trained on over 100,000 medical images from seven public datasets covering 24 organ categories, FLanS outperforms baselines in both in-domain and out-of-domain tests.
weaknesses: ## 1. Insufficient Motivation for FLanS:
The authors propose FLanS, a model designed to perform organ segmentation in CT scans based on free-form text prompts. However, the motivation behind this approach appears weak for the following reasons:

(1) Existing Tools: The authors claim that in real-world scenarios, clinicians often rely on natural language commands (e.g., "Highlight the right kidney" or "Segment the largest organ"). However, tools like TotalSegmentator[1,2,3] already segment most organs in CT and MRI scans effectively. These tools are widely integrated into platforms such as 3D Slicer[4], allowing clinicians to obtain segmentation results without needing text-based interactions.

(2) Questioning the Need: Given that TotalSegmentator covers a vast range of anatomical structures, the need for a text-based interaction model seems redundant. The authors do not provide a clear explanation of what organs or structures remain uncovered by existing solutions.

## 2. Addressable Challenges Without FLanS
The authors identify another challenge related to orientation differences between preprocessed datasets:

Orientation Standardization: The authors argue that varying scan orientations pose a problem that requires a "Canonicalized Network." However, this issue can be readily addressed with widely available Python libraries such as SimpleITK and nibabel, which can standardize 3D medical image orientations. Training an additional network specifically for orientation seems unnecessary and overcomplicated.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Comparison to the state-of-the-art baselines from natural images in the tasks like RES(Referring Expression Segmentation)[1]. The paper claims superior performance over SOTA baselines. However, it would be beneficial to see more direct comparisons, including visual examples and error analysis, to better understand where and how FLanS outperforms existing methods.
2. Lack of theoretical analysis, as a paper submitted to ICLR.
3.More details of making such a dataset.


[1] Liu, Chang, Henghui Ding, and Xudong Jiang. "GRES: Generalized referring expression segmentation."Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
soundness: 3
strengths: FLanS is trained on over 100k images and demonstrates strong language understanding and segmentation accuracy across various datasets. The key contributions are:
A RAG-based generator for diverse text prompts.
The FLanS model for text-driven medical image segmentation.
A symmetry-aware module for consistent segmentation across different scan orientations.
summary: The paper presents FLanS, a novel medical image segmentation model that can understand and respond to free-form text prompts. It features a RAG-based text prompt generator and a symmetry-aware canonicalization module.
weaknesses: 1. Comparison to the state-of-the-art baselines from natural images in the tasks like RES(Referring Expression Segmentation)[1]. The paper claims superior performance over SOTA baselines. However, it would be beneficial to see more direct comparisons, including visual examples and error analysis, to better understand where and how FLanS outperforms existing methods.
2. Lack of theoretical analysis, as a paper submitted to ICLR.
3.More details of making such a dataset.


[1] Liu, Chang, Henghui Ding, and Xudong Jiang. "GRES: Generalized referring expression segmentation."Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.