Pancakes: Consistent Multi-Protocol Image Segmentation Across Biomedical Domains
Authors: Marianne Rakic
Siyu Gai
Etienne Chollet
John Guttag
Adrian V Dalca
Venue: NeurIPS 2025 poster ()
Keywords: Segmentation
Medical imaging
Foundation model
Primary area: machine_learning_for_sciences
TL;DR: A new segmentation framework and sampling mechanism to produce multiple consistent label maps across similar images on held-out datasets, outperforming baselines by a wide margin.

Abstract: A single biomedical image can be segmented in multiple valid ways, depending on the application. For instance, a brain MRI may be divided according to tissue types, vascular territories, broad anatomical regions, fine-grained anatomy, or pathology. Existing automatic segmentation models typically either (1) support only a single protocol---the one they were trained on---or (2) require labor-intensive prompting to specify the desired segmentation. We introduce _Pancakes_, a framework that, given a new image from a previously unseen domain, automatically generates multi-label segmentation maps for _multiple_ plausible protocols, while maintaining semantic consistency across related images. In extensive experiments across seven previously unseen domains, _Pancakes_ consistently outperforms strong baselines, often by a wide margin, demonstrating its ability to produce diverse yet coherent segmentation maps on unseen domains.

Pancakes: Consistent Multi-Protocol Image
Segmentation Across Biomedical Domains
Marianne Rakic MIT CSAIL, MGH mrakic@mite edu
Siyu Gai MIT CSAIL, MGH
Etienne Chollet MIT CSAIL, MGH
John V. Guttag MIT CSAIL
Adrian V. Dalca MIT CSAIL, HMS, MGH
Protecol 1:
Image
Tissue Type
Subcortical Structures
Cortical and subcortical structures
MOL
Figure Examples of different expert provided protocols for brain MRL Any biomedical image can be segmented in many different ways. For example, protocol here corresponds to coarse-grained categorization of tissue types. Colors correspond to distinct ROIs (the choice of colors is arbitrary). Typical neural networks follow fixed protocol, specified explicitly or implicitly by the user.
Abstract
A single biomedical image can be meaningfully segmented in multiple ways, depending on the desired application. For instance, brain MRI can be segmented according to tissue types, vascular territories, broad anatomical regions, fine- grained anatomy,
pathology, etc. Existing automatic segmentation models typically either (1) support only single protocol the one they were trained on or (2) require labor-intensive manual prompting to specify the desired segmentation.
We introduce Pancakes, a framework that, given a new image from previously unseen domain, automatically generates multi-label segmentation maps for multiple plausible protocols, while maintaining semantic consistency across related images. Pancakes introduces new problem formulation that is not currently attainable by existing foundation models. In series of experiments datasets, we demonstrate that our model can significantly outperform existing foundation models in producing several plausible whole-image segmentations, that are semantically coherent across images.
seven held-out
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
There are many ways to segment biomedical image. Depending on their goals, clinicians or biomedical researchers employ a specific segmentation protocol, which defines the regions of interest (ROIs) to be segmented (Figure I]. This could involve, for example, segmenting major anatomical classes, granular anatomical regions, diffuse tissue types, systemic structures like vessels or nerves, pathologies, functional areas 66 28
45
63 80 [19].
Existing learning-basee based biomedical image segmentation tools require specification of a segmentation 130 Fully-automated models are trained to segment an image using Recent in-
protocol 16 58
16
123
training pairs, which implicitly define the protocol [8 48 99
context or few-shot models also use image-segmentation pairs, but take them inpul to specify the desired segmentation protocol for target image [8] [6
96 [17]. Interactive models rely on user interactions to indicate the desired ROIs [58] [16 122 123 In all of these strategies, the desired
segmentation protocol is specified by the user (interactively or by example). This is a substantial burden on biomedical researchers, who commonly need to segment a new biomedical image with potentially new segmentation protocol 4 [6].
Our goal is to support new capability: enabling biomedical researchers and clinicians to explore a diverse set of plausible, semantically consistent segmentations for a previously unseen collection of images. We propose fundamentally new approach to segmenting a new biomedical dataset. Instead of requiring the user to specify the protocol for new task, our method, Pancakes, produces segmentation maps for multiple plausible protocols simultaneously, each consistent across images. After Pancakes has generated the label maps, a researcher or clinician can select which of the proposed protocols best aligns with their intended downstream use (e.g., anatomical volume analysis). We envision at least two broad classes of use:
(1) Rapid segmentation for protocols. New protocols are frequently introduced || 36 85
and there is a need to produce corresponding segmentations. If a scientist has a particular protocol in mind, but there are existing tools for segmenting it, they can choose the protocol from Pancakes that best aligns with their intended use.
(2) Exploratory population analysis. Pancakes will support users in discovering or selecting segmentation strategies appropriate for their scientific or clinical questions. For example, a clinical scientist who studies how anatomy relates to some outcome (e.g., progression of a disease) or predictor (e.g., genetics) can use Pancakes to quickly extract segmentations of multiple candidate anatomical regions that have never been segmented, compute their volumes, and test correlations with clinical outcomes thereby identifying promising candidate regions.
Pancakes takes image as input and produces distribution over segmentation protocols. It then uses a segmentation sampling mechanism to produce several complete, multi-label segmentation maps from diverse protocols for that image. Importantly, within chosen protocol, segmentation maps are semantically consistent across subjects a specific label denotes the same anatomical structure in every image of the collection.
In a series of experiments, we demonstrate that our model can significantly outperform baselines in producing several plausible whole- segmentations that semantically coherent across images. We show that Pancakes outperforms foundation segmentation models by a wide margin on seven held-out datasets.
2 Related work
Single-protocol biomedical image segmentation. Most existing biomedical image segmentation
models 18 48 99 are by design constrained specific biomedical domain and image type. For
example, some models specialize in segmenting images of brains [13] (28 38 77 [92]. hearts [112)
134], or eye vessels [50] 770 100). Each model learns specific segmentation protocol defined by the image-segmentation pairs used during training.
Universal segmentation. Recent universal models can each segment wide variety structures across biomedical domains. They are trained jointly large data collections containing diverse

---

Image Set segment
SAM secting!
SAM setting
UnSAM
Figure 2: Current automatic segmentation foundation models produce inconsistent segmen- tations. Given a set of similar images to segment (left), automatic foundation models can fully parcellate each image, but the obtained segmentations are not semantically consistent across images. Even in rare cases where the same structure is labeled on two images, the label index (color-coded here) is usually inconsistent. There is then clear mapping between the segmentations from the two samples, making it difficult for biomedical clinicians and researchers to use the results.
structures and image types, both for natural 8 58 97
115 117 and medical imaging 16
96
123
[24 130 [31] Some methods generalize to tasks by enabling condition, or prompt, input.
This conditioning could involve example mmage-segmentatinn
pairs 8
16 30
96
TT3
1
122],
user interactions such bounding boxes, clicks, or scribbles (58 76
97
12
135 or even
text [4 130 131 135). Providing this conditioning is labor-intensive, especially when tackling a new segmentation task involving large collection of images.
We also train Pancakes using large image collections, to generalize to new domains. However, we avoid the need for the user to laboriously prescribe the protocol and instead automatically estimate several segmentation maps from several plausible protocols.
Some universal models can completely partition an image from a new biomedical domain into multiple labels [58 68 116.. As we show in our experiments and illustrate in Figure 2 these segmentation maps are semantically inconsistent across subjects, with the same label having different meanings across images. In the rare case when the same structure is segmented in two images, the assigned label index is usually different, and establishing the correspondence is non-trivial.
Multi-protocol segmentation. Motivated by the fact that many objects in an image can be divided into subparts, some methods produce labels from hierarchy of protocols in an image [24] 88 114 [25]. This restricts the types segmentations produced to fixed protocols that are inherently hicrarchical. The methods trained on limited domains or specialized to natural images. In either case, they require the hierarchy to be explicitly provided, which makes them less broadly applicable. In contrast, our approach produces label maps from multiple protocols that need not be hierarchically related, while generalizing to unseen structures.
Ambiguity and uncertainty. Even within a well-defined protocol, many segmentation tasks and biomedical images involve substantial ambiguity. This can be caused by problems with the image acquisition (e.g., noise or low contrast), ambiguous definitions of the desired region, the down- stream goals following the segmentation step. Recent models capture variability among manual raters [96] 102 M. often by aggregating multiple predictions for given structure or protocol to obtain an uncertainty estimate (23). In our work, we jointly capture the ambiguity of the possible protocol and the inherent ambiguity in the image, but focus on the ability of single framework to produce segmentations that represent different protocols consistently across scans.
Deep-learning and sampling mechanisms. Deep-lcarning segmentation frameworks that produce different outputs for given input use implicit explicit mechanism to sample different solutions, such as variational autoencoders 10 [59 [60] diffusion models [95] 120 121 [26], multivariate Gaussian [86). in-context stochastic models [96). We build on these methods, and propose new mechanism to sample different segmentation protocols that are varied but consistent among images from the same domain.
Method
Given an image we let ym be multi-label segmentation map for specific protocol m composed of non-overlapping labels. Typically, a segmentation model go (æ)) = ÿm follows a fixed predefined protocol m.
3

---

Label Maps predictions She for surl bor in ser (s:)
Sampler Rg(3)
varisble: Yn To sumple protocols
Representtion Pest
Map m
Reduaee Repear
Operations
Map
Subjects in set (x)
Distribusion Estimaner 1
Interaation betweeen e sample 9a
Mag M
Figure 3: Method Schematic. To produce multiple consistent label maps for an image set {r,). we first estimate the distribution parameters & via foy (z)) We then sample from the distribution with parameters & through the random variable r'= ha, (Tm ②) = 9m.
Instead, we design framework that produce set of label maps (ym for M different protocols, summarized Figure 3 We estimate high-dimensional parameters 6 of a distribution P(ym: σ|2) over segmentation maps ymm spanning different segmentation protocols m. using the function fn, (z) = ø. We model the distribution parameters ø as a vector for every image pixel, encoding the likelihood of different labels that pixel location across protocols.
We model (x) Φ as a neural network with UNet architecture, which takes in an image and outputs the parameters Φb Below, we describe the mechanism for sampling segmentation maps from the distribution p(ym: Ø|x). We then define a loss that encourages segmentation maps for given protocol m to be semantically consistent across set of S images from the same biomedical domain.
Protocol sampling. We design a new mechanism to produce diverse segmentation maps across different protocols Vm p(ym: 6|x). Let random integer I' (M, K) U(1, M x U(1, Km where is the discrete uniform distribution, M is a maximum number of label maps and Kmaa maximum number of labels for per map. We compute label map he, (é. rmm Dm given a deterministic function he.- The function first forms an intermediate representation em = e(rm), building concepts from positional embedding. We concatenate this representation with the distribution parameters at each image location, and use a shallow fully-convolutional network to yield the final segmentation maps:
Im = ha, (øllum)
ha, (Jo, (e))))
(2)
We predict the distributional parameters & once, and then efficiently compute ha, (o]le([rm)) })) } for a set of random integers {-m}-
The intermediate representations Um = (mm capture all labels & in protocol m. Let thy and uy be
vector representations corresponding to protocol m and label kt inspired by position embedding [13]. We model A ummll Mm.l/u.. where denotes the concatenation operation. Specifically, given integer values f, we form vector representation ut as:
2j sin (2,22) + 7)=
2j+1 sin(2r,23
T 2
(3)
with
t
21 where uf denotes entry j in vector We, == -=1,....... 1,...,, and is hyperparameter determining the size of the vector. We use this formulation to form vectors for any desired um and üg. which, in turn, enables us form Um for any given protocol. Specifically, the periods T for both vector representations t.m and are determined by the random variable Tm. i.c. the values of M and K sampled.
T 2 T
Inference. During inference, Pancakes is given set of images {xx} as input. For each image in the set, Pancakes produces a variable number of label maps fs.m.k from various protocols with a variable number of labels i semantically consistent across the images.
3.1 Training strategy
Our goal enable off-the-shelf multi-protoco, segmentation of any biomedical image x, especially for those not seen during training. To achieve this, train single Pancakes model on wide

---

Sct
Set
Set
Image Sets (Model Imput)
Binary Label
ysmk
Figure 4: Example input and binary label available at training, Images from the same set come from the same domain.
array of biomedical datasets spanning diverse domains and segmentation protocols. In most realistic scenarios, only a subset of the labels (often one label) in protocol are available each dataset, and most often only one protocol. At each training iteration, collection. If a dataset contains multiple protocols (which is rarely available in public data), we sample a protocol, and then sample a specific label task t within that protocol. Finally, randomly sample a set of images from that dataset, with that label segmented.
first sample dataset from the biomedical
At each iteration, for set of images {x,} } with associated ground-truie truth binary labels {ys}. sample M protocols containing at most labels each, and predict label maps {0m. for all M protocols.
Loss function. We design loss function that encourages the label maps of any produced protocol to be consistent across the image set. The loss function also enables learning from data with only a subset of labels segmented in each protocol, and encourages diversity of predicted candidate protocols. We develop this further in the Supplementary Section B
We define dmm . m,,,,, {2+}) EE, Dice.emmemmm V/3] ]] as the average Dice score for protocol m and label I across the samples s, where Vm.ssk is the binary map of label k of prediction @m... Denoting T as the set of all possible tasks and S the set of all image sets, optimize model parameters B and 0, by minimizing the loss function
C(0,, On. T) EE (WDam V.)l
(5)
with Leeg (10. = min dm,,k ( (1damm.) {v(]).
(6)
mnkk
By only penalizing the segmentation of the best performing predicted protocol m and label k the loss function encourages diverse label map samples Um. at least one candidate segmentation map matches the ground-truth binary label (58 96). By averaging the loss terms across the set, loss encourages label k of protocol m refer to the same region in each image. Figure 4 shows examples of the inputs and their corresponding binary labels used in the loss.
Augmentations. We apply standard data augmentations to improve generalization, such Gaussian noise, blur, contrast changes, affine, and elastic transforms. Building on recent strategies [16], we distinguish between in-task augmentations are applied independently to each element set and aim to increase the diversity of sets. The task augmentations aid in increasing the number of protocols available at training. A complete list of augmentations and parameters provided in the supplemental material in F3
types of augmentations, in-task augmentations and task augmentations. The
applied consistently across elements of the same and
Synthetic Data. To improve the generalization to new domains, we use synthetic data 13 build on Anatomix (27. First, we create maps by sampling binary segmentations from TotalSeg- mentator [118]. To simulate images from the same set representing the same biomedical region, we first sample a single label map, which is shared across the elements one set. We then create label maps by applying affine and elastic transforms to the original label map independently. We assign an intensity to all pixels given label, and apply several related augmentations to create a synthetic corresponding image set.
16
5

---

3.2 Implementation details
For function fo, (.), we use a UNet-like architecture, with convolutional layers of 32 features followed by PReLU activation [9] The function he, (-) is a series of convolution layers with skip connections. We use the SoftMax function across the dimension to obtain multi-label segmentation maps, with non-overlapping labels. This ensures that our protocols are complete, assigning label to each image pixel, instead of partial, assigning labels to only specific regions of the image. During training, we sample the maximum number of labels K, the number of protocols M, and the set size S uniformly from a fixed range: K E [5, 40], M E [5, 15J, S E [2, 5]. Within the same batch, different sets come from different domains. At inference time, K and M can be chosen by the user, while S is determined by the number of images in the set to segment. We use the Adam W optimizer [74] with a learning rate of 0.0001 [56].
4 Experiments
We evaluate Pancakes on a broad battery of biomedical segmentation tasks, with three goals: (1) verify that each protocol generated by the model semantically consistent across images; (2) quantify how well Pancakes matches manual segmentations of provided protocol; and (3) study how design choices during training and inference affect diversity and accuracy. We pay special attention to datasets and imaging domains unseen during training.
Evaluation. The evaluation of a method's ability to produce segmentation maps for different plausible protocols, consistently across subjects, is substantially more challenging than evaluation in standard (fixed protocol) segmentation tasks. In most test datasets, only one protocol is provided and only one label is available from that protocol.
We start by assessing if any of the predicted protocols produce label that closely matches a ground- truth label. We then capture whether this label is consistent across the set. Specifically, we compute the Dice score [107] [09] between the available ground-truth 3 and the labels produced for each label map and each image in the set De.m,k- We then compute the average across subjects in the set: E, { (Dice(De me.m.k k. v.)]]. Finally, we record the produced label that performs best, identified by a specific map m and label k. We call this Set Dice:
max E,, [Dice(()emm.))) v.)). m,k
This metric captures: (1) Consistency: a given label should represent the same region in all images in a set; and (2) Accuracy: for each individual image, how well does the best label match the ground-truth segmentation?
Because our goal is to generate a family of anatomically coherent multi-protocol label maps, no single user-provided mask can serve as a universal "ground-truth". Therefore, overlap metries such as Dice score offer only a partial assessment. We complement our quantitative evaluation with visualizations that illustrate the range of protocols generated by Pancakes. We assess these examples in two aspects: (1) each protocol should be consistent across subjects; and (2) different protocols should generate different, yet still anatomically plausible, partitions. Inevitably, some protocols will resonate more with readers' expectations than others. Our goal in this visualization is to demonstrate the diversity and semantic coherence Pancakes can achieve.
Data. We train Pancakes a large, diverse collection of biomedical data, and evaluate the multi-
protocol segmentations produced on images from held-out datasets. We use Megamedical [16]
96
123), which covers many biomedical domains 22 3 6
12
14
16
19
22 29
31
32
35
37
41
42
47
49 51
52
55, 61 67 69 (15 78 82 87 93 94,
98
101
103 LOS 106 088
10
127
129
132
133 This dataset has diverse anatomies such as thoracic organs (75), brain 80 eye 78 :
and diverse modalities including XRay 47,. CT (75,, ultrasound (66] and fundus images 46 We partition this collection into three subgroups. Training datasets are used at training time, including model weight optimization and backpropagation. A complete list of the training datasets can be found Table 4 in the supplemental material. Development datasets, ACDC [[I], PanDental [2], and Sebn [132] are not used in training but are to evaluate models during development. Held-owt datasets. These datasets are only used for final evaluation. They include QUBIQ Prostate dataset 83
SCD 194], WBC [33). BUID 33. LIDC-IDRI (5), DDTI [90], and STARE [46).

---

90
Pancakes
80
SAM
ScribblePrompt
70
UnSAM
MedSAM
60
Set Dice
50
40
30
20
10
BUID
DDTI
HipXRay
QubiqTk1
QubiqTk2
SCD
STARE
WBC
LIDC
Figure 5: Pancakes generalizes well to unseen datasets. We evaluate Pancakes and baselines with the Set Dice metric.
The images within each dataset are also split into training, validation, and test splits. We train the models the training split of the training datasets. We used the training split of the development datasets to monitor out-of-diss of-distribution capabilities. We report results on the test splits both the development (in the supplemental material) and held-out datasets. We split the dataset based subjects, and ensured that there was no train/validationntesl subject cross-contamination.
We also synthesized 120, 000 training image-segmentation pairs as described in Section We refer to these examples as Anatomix data.
Benchmarks. To our knowledge, there are methods that attempt the same task Pancakes. We compare our work to four methods that can be used to produce segmentation maps for new images, but none of these were designed to produce multiple protocols consistently across subjects.
SAM [58, 97] the Segment Anything Model (SAM) is an interactive image segmentation model trained mostly on natural images. SAM involves whole-image-segmentaiic a grid of simulated clicks over the whole image, removes high-overlapping regions and selects most likely masks using confidence threshold. We use this mode, and produce diverse whole-image segmentations by varying the confidence threshold.
mode, which produces
ScribblePrompr 123
ScribblePrompt (SP), is an interactive segmentation tool trained on Megamed- ical. We use the SP-SAM, which performs best on clicks, and obtain coltipproooool and multi-label using the same whole-image strategy we used for SAM.
trained specifically medical data, this model uses SAM-like architecture and is optimized for bounding box interactions. We produce whole-image segmentations using the SAM whole-image segmentation mode.
MedSAM [6]
UnSAM [16): trained on a curated set of natural images, this model produces segmentation masks using DINO (20) backbone (pretrained ResNet50 [40]) and the Mask2Former (21) mask decoder. UnSAM produces a list labels to serve as label maps. We use the UnSAM+ model version, trained on part of the SA- IB dataset [58). To produce segmentations with diverse protocols and labels, use the UnSAM's existing whole-image segmentation scheme.
Results
For our main evaluation, error bars are 95% confidence interval using 1000 bootstraps of all sets. Additional per-dataset performance and ablations shown in the supplemental material.

---

IBC
HipXRay
Qubiq
Figure Pancakes set consistency. For the same protocol, each label, identified by color, is assigned to similar structures across set of images.
Fixed set size und labels K
First label map output as K inereases
WBC
K-10
Qubiq HipXRay
K-10
K-8
0
3
5
10
15
20
25
Label map ID,
Max Labels first protocol ID (m-0)
Figure 7: Effeet of and K. Left: Input image for the first subject the set (s = 0) with set size S = 3. Middle: Pancakes can produce diverse label maps with fixed K labels per protocol. Right: Increasing the maximum number of labels K tends to lead to finer structures in the produced label maps.
Quantitatively, Figure 5 shows that Pancakes outperforms the baselines on all unseen datasets, often by margin of more than 20 Dice points. For a more detailed assessment, we separately report accuracy, consistency, and the effects of hyperparameters, and visualize results in several scenarios.
Accuracy versus consistency. Figure S shows that Pancakes performs well on both segmentation accuracy and semantic consistency across images. For individual accuracy (set size S 1), which does not penalize semantic consistency across images, Pancakes performs similarly to SAM, and is superior all the other baselines. As the set size S increases, all baselines fail to produce semantically consistent segmentations, while Pancakes segmentations remain consistent across the set, leading
70
60
Set Dice
50
40
MedSAM
30
UnSAM
ScribblePrompt
20
SAM
Pancakes
10
2
3
Set Size
Figure 8: Pancakes is both consistent and accurate compared to baselines. When evaluated solely on accuracy 1), Pancakes is comparable to SAM and outperforms baselines. As S increases, Pancakes is the only model whose performance is not degraded.

---

2
53
54
54
Table 1: Inference time and number of parameters. Pancakes has substantially fewer parameters and is sig- nificantly faster than baseline methods.
4
56
56
58
57
58
M
65
60
60
#Param
S=1 (sec.)
S=3 (sec.)
Ours
0.22 M
0.10 + 0.04
0.12 + 0.03
16
59
59
65
64
62
SAM
641M
3.13:1.116
2.94 + 0.24
SP
93.7M
1.99 0.13
1.85 + 0.18
20
30
40
K
MedSAM
93.7M
2.12
1.94 + 0.18
UnSAM
23M
0.012
0.45 = 0.03
Figure 9: Influence of segmentation maps M and labels K on segmenta- tion quality for 5 =3
to steady Set Dice. We hypothesize that SAM is better than biomedical baselines because it was trained on a wider variety labels and images. Therefore, SAM less prone to mnsk-overi. In the Supplementary Section[]] we also report results for various M and K, and also evaluate using the Set Surface Distance and Set loU metrics.
Figure 6 iillustrates Pancakes predictions for fixed protocol, and highlights visually the semantic consistency across the images set. Additional visualizations with the baselines are shown in Section
Overall, Pancakes outperforms matches the other methods in producing plausible protocols, and outperforms all methods by substantial margin in producing semantically consistent protocols.
Influence of M and K. Figure 7 jillustrates the diversity of protocols captured in Pancakes segmenta- tion outputs. They are produced by fixing the set size S and the maximum number of labels K and performing one forward pass through the network. The label maps are different from one another across protocol ID and maximum labels K, capturing structures at various granularity levels. 7 also shows that increasing the number of labels K per protocol results in segmentations of finer structures.
Figure
Figure captures the quantitative effects of the number of label maps M and maximum label number K for a fixed set size S. Producing more protocols leads to better performance in general, while performance function of is more variable. We hypothesize that this arises from the non- overlapping nature of labels within each protocol. Ambiguous regions that could belong to multiple labels require separate protocols represent each plausible interpretation, having more protocols enables the model to better capture such ambiguities.
Efficiency. We study runtime requirements by running the models on the HipXRay [37 test split.
the average run time across 1000 trials. Pancakes is substantially faster than all baselines, because its efficient fully-convolutionall architecture, leading to leaner model and fewer parameters.
Analysis. We use Ser Dice with 5 = 3, and use M and K = 20 for Pancakes. As supplementary Figure 15 shows, performance the development datasets saturates M = 8 and K 20, with only marginal gains achieved by further raising M. We therefore chose to use these parameters as a balance between performance and the number of structures clinical users might actually expect in practice.
Influence of synthetic data. We compare versions Pancakes trained with real data only, synthetic data only, or both. When trained with both real and synthetic data, Pancakes consistently outperforms the other variants Dice score (p 0.05 with a paired Student-t test), for M 16, as shown in Table 2
We find that for M 8, the performance change is not statistically significant.
Pancakes and interactive segmentation. user has particular segmentation protocol in mind, but there no existing tools for it, they choose the protocol from Pancakes' outputs that best

---

Interactive sogmentation
0.8
Table 2: Influence synthetic data across set size. For M 16, training with both synthetic data and Megamedical yields a small yet consistent improvement over only training with Megamedical.
0.7
0.6
with Fancakes Ink Without Parcales Init
Both
Megamedical
Synthetic
10
==
73.2
5.5
71.1 = 6.2
56.3 4.9
Figure 10: Interactive Segmentation with Pancakes initialization. Pancakes can be used as an initialization for inter- active segmentation to obtain high qual- ity labels faster.
S=2
67.3
6.0
65.8 + 6.7
45.8 5.0
==3
67.4
6.0
65.7 + 6.8
44.3 5.4
S=5 68.4
5.6
67.4 + 6.9
42.7 + 5.5
aligns with their intended use. This choice will often suffice. In cases where Pancakes' segmentations are not sufficiently accurate-for example, when targets are far from the training distribution-the segmentation maps can offer excellent initializations to interactive segmentation systems such as ScribblePrompt [123]. To demonstrate this, we compared using ScribblePrompt with and without initialization using Pancakes' predictions: (I) on average, Pancakes' predictions can be improved by Dice points with a single interactive click, for set sizes larger than one. (2) on average, using ScribblePrompt with a Puncakes-initiallin by half. If used alone. it takes ScribblePrompt 5 to 8 clicks for the prediction quality to plateau, while, when initialized with Pancakes' predictions, ScribblePrompt can reach the same results in clicks.
segmentation reduces the number of required interactions
6 Assumptions & Social Impact
In this work, we made several core assumptions. First, we assume that the user has an idea of the desired labels and can select few K values a priori. Pancakes is intended for biomedical experts. We assume that as they use the tool, they can identify the mapping between a label and a known structure. Second, assume that visualizing several protocols simultancously reasonable. Third, we assume that while we trained and evaluated on a diverse set of medical images, we likely did not capture all biomedical image types and domains that user might encounter. Fourth, Pancakes intended to replace existing clinically validated segmentation protocols. If there is an existing tool for a particular protocol, we would advise using
We aim for this work to inspire a new approach to using foundation models in biomedical imaging. Pancakes is designed to be efficient and accessible, especially in resource-constrained settings. It can support both data annotation and exploratory analysis, as well as serve as tool for developing future foundation models. However, this version is intended for research collection of small biomedical datasets, we have not evaluated it for potential societal biases.
only. While trained broad
7 Conclusion
We introduced Pancakes, a new framework that predicts segmentation maps for multiple protocols in previously unseen biomedical imaging domains, with each protocol being semantically consistent across images. Pancakes estimates a distribution over plausible label map protocols and provides a mechanism to sample multiple segmentation maps from this distribution.
Predicting plausible segmentations in a new domain, without prior knowledge of the number of labels or their associated shapes, is a challenging task. Our experiments demonstrate that Pancakes achieves tate-of-the-an performance on seven held-out datasets. We believe this work addresses an important, previously unaddressed problem, enabling biomedical researchers to segment entire collections of images without requiring manual annotations or example segmentations. This can, in turn, substantially speed up downstream biomedical studies.
10

---

8 Acknowledgement
We would like thank Neel Dey for the very helpful discussions and feedback. This research was supported by the National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award number RO1EB033773, the Eric and Wendy Schmidt Center the Broad Institute of MIT and Harvard, Quanta Computer Inc. Some of the computation resources required for this research was performed on computational hardware generously provided by the Massachusetts Life Sciences Center.
References
[1] Topbrain 2025 grand challenge. cessed: 2025-08-23.
Ac-
[2] A. H. Abdi, S. Kasaei, and M. Mehdizadeh. Automatic segmentationn mandible in panoramic x-ray. Journal of Medical Imaging, 2(4):044003, 2015.
[3] W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy. Dataset of breast ultrasound images.
Data Brief. 28: 104863, 2020.
[4] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Sopp-Schneider B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers, et al. The medical segmentation decathlon. Nature
communications, 13(1):4128, 2022.
[5] S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. Meyer, A. P. Reeves, B. Zhao, D. R. Aberle, C. Henschke, E. A. Hoffman, et al. The lung image database consortium (lide) and image database resource initiative (idri): completed reference database of lung nodules on et scans. Medical physics, 38(2):915 931, 2011.
[6] U. Baid, S. Ghodasara, S. Mohan, M. Bilello, E. Calabrese, E. Colak, K. Farahani. J. Kalpathy- Cramer, C. Kitamura, Pati, et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107. 023112 2021.
[7] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Fara- hani, and C. Davatzikos. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data, 111::113,, 2017.
[8] I. Balazevic, D. Steiner, N. Parthasarathy, R. Arandjelović, and O. Henaff. Towards in-context scene understanding, Advances in Neural Information Processing Systems, 36, 2024.
[9] S. Bano, F. Vasconcelos, L. M. Shepherd, E. Vander Poorten, T. Vercauteren, S. Ourselin, L. David, J. Deprest, and D. Stoyanov. Deep placental vessel segmentation for fetoscopic mosaicking. In Medical Image Computing and Computer Assisted Intervention 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part III 23, pages 763-773. Springer, 2020.
[10] F. Baumggrtner, K. C. Tezcan, K. Chaitanya, A. M. Hötker, U. Muchlematter, K. Schawkat, A. S. Becker, O. Donati, and E. Konukoglu. Phiseg: Capturing uncertainty in medical image segmentation. In Medical Image Computing and Computer Assisted Intervention- 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part 11 22, pages 119-127. Springer, 2019.
[11] O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, X. Yang, P.-A. Heng. Cetin, K. Lekadir, O. Camara, M. A. G. Ballester, et al. Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging, 37(11):2514-255 2018.
[12] P. Bilic, P. Christ, E. Vorontsov, Chlebus, H. Chen, Q. Dou, C.-W. Fu, X. Han, P.- A. Heng, J. Hesser, et al. The liver tumor segmentation benchmark (lits). arXiv preprins arXiv:1901. 04056, 2019.
[13] B. Billot, D. N. Greve, O. Puonti, A. Thielscher, K.
Van Leemput, B. Fischl, A. V. Dalca, J. F. Iglesias, et al. Synthseg: Segmentation of brain mri scans of any contrast and resolution without retraining. Medical image analysis, 86:102789, 2023.
[14] N. Bloch, A. Madabhushi, H. Huisman, J. Freymann, J. Kirby, M. Grauer, A. Enquobahrie, Jaffe, L. Clarke, and K. Farahani. Nei-ishi 2013 challenge: automated segmentation of prostate structures. The Cancer Imaging Archive, 370(6):5, 2015.
11

---

[15] M. Buda, A. Saha, and M. A. Mazurowski. Association of genomic subtypes of lower- grade gliomas with shape features automatically extracted by deep learning algorithm. Computers in biology and medicine, (09:218-223 2019.
[16] V. I. Butoi, J. J. G. Ortiz, T. Ma, M. R. Sabuncu, J. Guttag, and A. V. Dalca. Universeg: Universal medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21438-21457 2023.
[17] J. C. Caicedo, A. Goodman, K. W. Karhohs, B. A. Cimini, Ackerman, M. Haghighi, C. Heng, T. Becker, M. Doan, C. McQuin, M. Rohban, S. Singh, and A. E. Carpenter. Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl. Nature Methods, 16(12):124712253, Dec. 2019.
[18] H. Cao, Y. Wang, J. Chen. D. Jiang, X. Zhang, Q. Tian, and M. Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. In European conference computer vision, pages 205-218. Springer, 2022.
[19] A. Cardon, S. Saalfeld, S. Preibisch, B. Schmid, A. Cheng, Pulokas, P. Tomancak, and V. Hartenstein. Isbi challenge: Segmentation of neuronal structures in em stacks.
[20] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in sulf-supervised vision transformers. In Proceedings of the IEEF/CVF interna- tional conference on computer vision, pages 9650-9660, 2021.
[21] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attentinn mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-1299, 2022.
[22] N. C. F. Codella, D. A. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. K. Mishra, H. Kittler, and A. Halpern. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (ISIC). CoRR, abs/17 10000006 2017.
[23] S. Czolbe, K. Arnavaz, O. Krause, and A. Feragen. Is segmentation uncertainty useful? In Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28-June 30, 2021, Proceedings 27, pages 715-726. Springer, 2021.
[24] D. de Geus, P. Meletis, C. Lu, X. Wen, and G. Dubbelman. Part-aware panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5485-5494, 2021.
[25] E. Decenciere, G. Cazuguel, X. Zhang, G. Thibault, J.-C. Klein, F. Meyer, B. Marcotegui, G. Quellec, M. Lamard, R. Danno, et al. Teleophta: Machine learning and image processing methods for teleophthalmology. Irbm, 34(2): 1e21-196-00 203, 2013.
[26] A. Degerli, M. Zabihi, S. Kiranyaz, T. Hamid, R. Mazhar, R. Hamila, and M. Gabbouj. Early detection of myocardial infarction in low-quality echocardiography. IEEE Access, 9:34442-34453, 2021.
[27] Dey, B. Billot, H. F. Wong, C. J. Wang, M. Ren, P. F. Grant, A. V. Dalca, and P. Golland. Learning general-purpose biomedical volume representations using randomized synthesis, 2024.
[28] B. Fischl. Freesurfer. Neuroimage, 62(2):774-781,2122
[29] J. Gamper, N. Koohbanani, K. Benes, S. Graham, M. Jahanifar, S. Khurram, A. Azam, K. Hewitt, and N. Rajpoot. Pannuke dataset extension, insights and baselines. arxiv. 2020 doi: 10.48550. ARXIV, 2003.
[30] Y. Gao, D. Liu, Z. Li, Y. Li, D. Chen, M. Zhou, and D. N. Metaxas. Show and segment: Uni- versal medical image segmentation via in-context learning. arXiv preprint arXiv:2503.13359 2025.
[31] S. Gerhard, J. Funke, J. Martel, A. Cardona, and R. Fetter. Segmented anisotropic ssTEM dataset of neural tissue. figshare, pages 0-0, 11 2013.
[32] R. L.. Gollub, M. Shoemaker, M. D. King, T. White, S. Ehrlich, S. R. Sponheim, V. P. Clark, J. A. Turner, B. A. Mueller, V. Magnotta, et al. The meic collection: a shared repository of multi-modal, multi-site brain image data from a clinical investigation of schizophrenia. Neuroinformatics 11:367-388, 2013.
12

---

[33] S. Gousias, A. D. Edwards, M. A. Rutherford, S. J. Counsell, J. V. Hajnal, D. Rueckert, and A. Hammers. Magnetic resonance imaging the newbom brain: manual segmentation of labelled atlases in term-born and preterm infants. Neuroinage, 62(3):149--1509 2012.
[34] S. Gousias, D. Rueckert, R. A. Heckemann, L. E. Dyet, J. P. Boardman, A. D. Edwards, and Hammers. Automatic segmentation of brain mris of 2-year-olds into 83 regions of interest. Neuroimage, 40(2):672-684 2008.
[35] S. Graham, Q. D. Vu, S. E. A. Raza, A. Azam, Y. W. Tsang, J. T. Kwak, and N. Rajpoot. Hover-net: Simultancous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis, 58: 101563, 2019.
[36] D.N. Greve, B. Billot, D. Cordero. A. Hoopes, M. Hoffmann, A. V. Dalca, B. Fischl, J. E. Iglesias, and J. C. Augustinack. A deep learning toolbox for automatic segmentation of subcortical limbic structures from mri images. Neuroimage, 244:1 18610, 2021.
[37] D. Gut. X-ray images of the hip joints. 1, July 2021. Publisher: Mendeley Data.
[38] A. Hatamizadeh, V. Nath, Y. Tang, D. Yang, H. R. Roth, and D. Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors mri images. International MICCAI brainlesion workshop. pages 272-284, Springer, 2021.
[39] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-leve performance on imagenet classification. Proceedings of the IEEE international conference computer vision. pages 1026-1034 2015.
[40] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[41] N. Heller, F. Isensee, K. H. Maier-Hein, X. Hou, C. Xie, F. Li, Y. Nan, G. Mu, Z. Lin, M. Han, et al. The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits19 challenge. Medical Image Analysis, page 101821, 2020.
[42] M. R. Hernandez Petzsche, E. de la Rosa, U. Hanning, R. Wiest, W. Valenzuela, M. Reyes, M. Meyer, S.-L. Liew, F. Kofler, I. Ezhov, et al. Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset. Scientific data, 9(1):662, 2022.
[43] M. Hoffmann, B. Billot, D. N. Greve, J. E. Iglesias, B. Fischl, and A. V. Dalca. Synthmorph: learning contrast-invariant registration without acquired images. IEEE transactions on medical imaging, 41(3):543-5588 2021.
[44] A. Hoopes, V. I. Butoi, J. V. Guttag, and A. V. Dalca. Voxelprompt: A vision-language agent for grounded medical image analysis. arXiv preprint 2tt::241000899,, 2024.
[45] A. Hoopes, M. Hoffmann, D. N. Greve, B. Fischl, J. Guttag, and A. V. Dalca. Learning the effect of registration hyperparameters with hypermorph. volume 1, pages 1-30, 2022.
[46] A. Hoover, Kouznetsova, and M. Goldbaum. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Transactions on Medical imaging, 19(3):203-25 2000000---0---02 2000.
[47] H. in the Loop. Teeth segmentation dataset.
[48] F. Isensee, F. Jaeger, S. A. Kohl, Petersen, and K. H. Maier-Hein. nnu-net: self- configuring method for deep barning-based biomedical image segmentation. Nature methods, 18(2):203-211 2021.
[49] Y. Ji, H. Bai, J. Yang, C. Ge, Y. Zhu, R. Zhang, Z. Li, Zhang, W. Ma, X. Wan, et Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprint arXiv:2206. 08023, 2022.
[50] Q. Jin, Z. Meng, T. D. Pham, Q. Chen, L. Wei, and R. Su. Dunet: deformable network for retinal vessel segmentation. Knowledge-Band Based Systems, 178:149 162, 2019.
[51] R. Karim, R. J. Housden, M. Balasubramaniam. Z. Chen, D. Perry, A. Uddin, Y. Al-Beyatti, Palkhi, Acheampong. S. Obom, et al. Evaluation of current algorithms for segmentation of scar tissue from late gadolinium enhancement cardiovascular magnetic resonance of the left atrium: an open-access grand challenge. Journal of Cardiovascular Magnetic Resonance, 15(1):1-17, 2013.
13

---

[52] A. E. Kavur, N. S. Gezer, M. Bany. S. Aslan, P.-H. Conze, V. Groza, D. D. Pham, S. Chatterjee, P. Ernst, S. Özkan, B. Baydar, D. Lachinov, S. Han, J. Pauli, F. Isensee, M. Perkonigg, R. Sathish, R. Rajan, D. Sheet, G. Dovletov, O. Speck, A. Nürnberger, K. H. Maier-Hein, G. Bozdaği Akar, G. Unal, O. Dicle, and M. A. Selver. CHAOS Challenge combined (CT-MR) healthy abdominal organ segmentation. Medical Image Analysis, 69:101950, Apr. 2021.
[53] A. E. Kavur, N. S. Gezer, M. Barış, S. Aslan, P.-H. Conze, V. Groza, D. D. Pham, S. Chatterjee,
P. Ernst, S. Özkan, B. Baydar, D. Lachinov, S. Han, J. Pauli, F, Isensee, M. Perkonigg, R. Sathish, R. Rajan, D. Sheet, G. Dovletov, O. Speck, A. Nürnberger, K. H. Maier-Hein,
G. Bozdaii Akar, G. Ünal, O. Dicle, and M. A. Selver. CHAOS Challenge combined (CT-MR) healthy abdominal organ segmentation. Medical Image Analysis, 69: 69:101950, 2021.
[54] A. E. Kavur, M. A. Selver, O. Dicle, M. Barış, and N. S. Gezer. CHAOS Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data. Apr. 2019.
[55] A. E. Kavur, M. A. Selver, O. Dicle, M. Bariş, and N. S. Gezer. CHAOS CCombined (CT-MR)
Healthy Abdominal Organ Segmentation Challenge Data, Apr. 2019.
[56] D. P. Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.690
2014.
[57] S. Kiranyaz, A. Degerli, T. Hamid, R. Mazhar, R. E. F. Ahmed, R. Abouhasera, M. Zabihi, J. Malik, R. Hamila, and M. Gabbouj. Left ventricular wall motion estimation by active polynomials for acute myocardial infarction detection. IEEE Access, 8:210301-21033 2020.
[58] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.Y Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[59] S. Kohl, B. Romera-Paredes, C. Meyer, J. De Fauw, J. R. Ledsam, K. MlierrHHei, S. Eslami, D. Jimenez Rezende, and O. Ronneberger. A probabilistic u-net for segmentation of ambiguous images. Advances in neural information processing systems, 31, 2018.
[60] S. A. Kohl, B. Romera-Paredes, K. H. Maier-Hein, D. J. Rezende, S. Eslami, P. Kohli, A. Zisserman, and O. Ronneherger. A hierarchical probabilistic u-net for modeling multi-scale ambiguities. arXiv preprint arXiv: 13077, 2019.
[61] M. Krönke, C. Eilers, D. Dimova, M. Köhler, G. Buschner, L. Schweiger, L. Konstantinidou, M. Makowski, Nagarajah, N. Navab, et al. Tracked 3d ultrasound and deep neural network- based thyroid segmentation reduce interobserver variability in thyroid volumetry. Plos one, 076))500, 2022.
[62] H. J. Kuijf, M. Biesbroek, De Bresser, R. Heinen, S. Andermatt, M. Bento, M. Berseth, M. Belyaev, M. J. Cardoso, A. Casamitjana, et al. Standardized assessment of automatic segmentation of white matter hyperintensities and results of the wmh segmentation challenge. IEEE transactions on medical imaging, 38(1 1)2256-2568, 2019.
[63] M. Halklisova-Murgaasoo Gousias, J. P. Boardman, M. A. Rutherford, A. D. Edwards, et al. A dynamic 4d probabilistic atlas of the developing brain. Neurolmage, 54(4):2750-2766 2011.
P. Aljabar, L. Srinivasan, S. J. Counsell, V. Doria, A. Serag, I. S.
[64] 7. Lambert, C. Petitjean, B. Dubray, and S. Kuan. Segthor: segmentation of thoracic organs risk in ct images. In 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA), pages 1-6. IEEE, 2020.
[65] B. Landman, Z. Xu, J. Igelsias, M. Styner, T. Langerak, and A. Klein. Miccai multi-atlas labeling beyond the cranial vault-workshop and challenge. In Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Shuult-Worksho Challenge, volume 5, page 12, 2015.
[66] S. Leclere, Smistad, J. Pedrosa, A. Østvik, F. Cervenansky, F. Espinosa, T. Espeland, E. A. R. Berg, P.-M. Jodoin, T. Grenier, et al. Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEEE transactions on medical imaging. 38(9):2198 2210, 2019.
[67] G. Lemaitre, R. Martí, J. Freixenet, J. C. Vilanova, P. M. Walker, and F. Meriaudeau. Computer- aided detection and diagnosis for prostate cancer based on mono and multi-parametric mri: a review. Computers in biology and medicine, 60:8-31, 2015.
14

---

[68] F. Li, H. Zhang. P. Sun. X. Zou, S. Liu, J. Yang, C. Li, L. Zhang, and J. Gao. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04767, 2023.
[69] M. Li, Y. Zhang. Z. Ji, K. Xie, S. Yuan, Q. Liu, and Q Chen. Ipn-v2 and octa-500: Methodology and dataset for retinal image segmentation. arXiv preprint wrXiv:2012.07261, 2020.
[70] P. Liskowski and K. Krawiee. Segmenting retinal blood vessels with deep neural networks. IEEE transactions on medical imaging. (5(11):2369-233) 23809, 2016.
[71] G. Litjens, R. Toth, W. van de Ven, C. Hoeks, S. Kerkstra, B. van Ginneken, G. Vincent,
G. Guillard. N. Birbeck, Zhang, et al. Evaluation of prostate segmentation algorithms for mri: the promise challenge. Medical image analysis, 18(2):359-373, 2014.
[72] V. Ljosa, K. L. Sokolnicki, and A. E. Carpenter. Annotated high-throughput microscopy image sets for validation. Nature methods, 9(7):637-637, 2012.
[73] M. T. Löffler, A. Sekuboyina, A. Jacob, A.-L. Grau, A. Scharr, M. El Husseiii, M. Kallweit, C. Zimmer, TT. Baum, and J. S. Kirschke. vertebral segmentation dataset with fracture grading. Radiology: Artificial Intelligence, 2(4))e190138, 2020.
[74] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint
0rXiv:1711.051011 2017.
[75] X. Luo, W. Liao, J. Xiao, T. Song, X. Zhang, K. Li, G. Wang, and S. Zhang. Word: Revisiting organs segmentation in the whole abdominal region. arXiv preprint arXiv:211 1121111.2220 1.02403, 2021.
[76] J. Ma, Y. He, F. Li, L.. Han, C. You, and B. Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
[77] Q. Ma, L. Li, E. C. Robinson, B. Kainz, and D. Rueckert. Weakly supervised learning of cortical surface reconstruction from segmentations. In International Conference on Medical Image Computing and Computer-Assiiter Intervention, pages 7766677.72 Springer, 2024.
[78] Y. Ma, H. Hao, J. Xie, H. Fu, J. Zhang, J. Yang. Z. Wang. J. Liu, Y. Zheng, and Y. Zhao. Rose: retinal oct-angiography vessel segmentation dataset and new model. IEEE Transactions on Medical Imaging, 40(3):928-93) 2021.
[79] J. A. Macdonald, 7. Zhu, Konkel, M. Mazurowski, W. Wiggins, and M. Bashir. Duke liver dataset (MRI) v2, Apr. 2023.
[80] D. S. Marcus. T. H. Wang, J. Parker, J. G. Csernansky, J. C. Morris, and R. L. Buckner. Open access series of imaging studies (oasis): cross- sectional mri data in young. middle aged, nondemented, and demented older adults. Journal of cognitive neuroscience, 19(9):1498-1500 2007.
[81] K. Marek, D. Jennings, S. Lasch, A. Siderowf, C. Tanner, T. Simuni, C. Coffey, K. Kieburtz, E. Flagg. S. Chowdhury, et al. The parkinson progression marker initiative (ppmi). Progress in neurobiology, 95(4):629 66352 2011.
[82] M. A. Mazurowski, K. Clark, N. M. Czarnek, P. Shamsesfandabadi K. B. Peters, and A. Saha. Radiogenomics of lower-grade glioma: algorithmically tumor shape is associated with tumor genomic subtypes and patient outcomes in a multi-instiuutional study with the cancer genome atlas data. Journal of neuro-oncology. 133:27-35, 2017.
[83] Menze, L. Joskowiez, S. Bakas, A. Jakab, E. Konukoglu, A. Becker, A. Simpson, and R. D. Quantification of uncertainties in biomedical image quantification 2021. 4th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), 2021.
[84] B. H. Menze, A. Jakab, S. Bauer, J. Salpathy-Cramer K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al. The multimodal brain tumor image segmentation benchmark 2014.
(brats). IEEE transactions on medical imaging, 34(10):1993-2000
[85] R. Merdietio Boedi, S. Shepherd, F. Oscandar, S. Mânica, and A. Franco. 3d segmentation of dental crown for volumetric age estimation with ebet imaging. International Journal of Legal Medicine, 137(1):123-130 2023.
[86] M. Monteiro, L. Le Folgoc, D. Coelho de Castro, N. Pawlowski, B. Marques, K. Kam- nitsas, M. van der Wilk, and B. Glocker. Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty. Advances in Neural Information Processing Systems, 111122566127677 2020.
15

---

[87] A. Montoya, Hasnin, kaggle446, shirzad, W. Cukierski, and yffud. Ultrasound nerve segmen- tation, 2016.
[88] J. Myers-Dean, J. Reynolds, B. Price, Y. Fan, and D. Gurari. Spin: Hierarchical segmentation with subpart granularity in natural images. arXiv preprint arXiv:2407.0960 2024.
[89] K. Payette, P. de Dumast, H. Kebiri, I. Ezhov, J. C. Paetzold, S. Shit, A. Iqbal, R. Khan, R. Kottke, P. Grehten, et al. An automatic multi-tissue human fetal brain segmentation benchmark using the fetal tissue annotation dataset. Scientific Data, 8(1):1-14, 2021.
[90] L. Pedraza, C. Vargas, F. Narváez, O. Durán, E. Muñoz, and E. Romero. An open access thyroid ultrasound image database. In 10th International symposium on medical information processing and analysis, volume 9287, pages 188-193. SPIE, 2015.
[91] L. Pedraza, C. Vargas, F. Narváez, O. Durán, E. Muñoz, and E. Romero. An open access thyroid ultrasound image database. In 10th International symposium on medical information processing and analysis, volume 9287, pages 183. SPIE, 2015.
[92] H. Peiris, M. Hayat, Z. Chen, G. Egan, and M. Harandi. A robust volumetric transformer for accurate 3d tumor segmentation. In International conference on medical image computing and computer-assistet intervention, pages 162-172. Springer, 2022.
[93] P. Porwal, S. Pachade, R. Kamble, M. Kokare, G. Deshmukh, V. Sahasrabuddhe, and F. Meri- audeau. Indian diabetic retinopathy image dataset (idrid), 2018.
[94] P. Radau, Y. Lu, K. Connelly, G. Paul, A. Dick, and G. Wright. Evaluation framework for algorithms segmenting short axis cardiac mri. The MIDAS Journal-Cardic MR Left Ventricle Segmentation Challenge, 49, 2009.
[95] A. Rahman, J. M. J. Valanarasu, I. Hacihaliloglu, and V. M. Patel. Ambiguous medical image segmentation using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11536-18 1 1546, 2023.
[96] M. Rakic, H. E. Wong, J. J. G. Ortiz, B. A. Cimini, J. V. Guttag, and A. V. Dalca. Ty- che: Stochastic in-context learning for medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1 11159-11173, 2024.
[97] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 00714, 2024.
[98] Rister, D. Yi, K. Shivakumar, T. Nobashi, and D. L. Rubin. CT-ORG, a new dataset for multiple organ segmentation in computed tomography. Scientific Data, 7(1):381, Nov. 2020.
[99] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted Intervention-MICOO 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234 241. Springer, 2015.
[100] F. Rossant, M. Badellino, A. Chavillon, I. Bloch, and M. Paques. A morphological approach for vessel segmentation in eye fundus images, with quantitative evaluation. Journal of Medical Imaging and Health Informatics, 1(1):22-4) 2011.
[101] A. Saporta, X. Gui, A. Agrawal, A. Pareek, S. Truong, C. Nguyen, V.-D. Ngo, Seekins, F. G. Blankenberg, A. Ng, et al. Deep learning saliency maps do not accurately highlight diagnostically relevant regions for medical image interpretation. MedRxiv, 2021.
[102] A. Schmidt, P. Morales Arraaee-Alvarez, and R. Molina. Probabilistic modeling of inter-and intra- observer variability in medical image segmentation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 21097-21106. 2023.
[103] C. Seibold, S. ReiB, S. Sarfraz, M. A. Fink, V. Mayer, Sellner, M. S. Kim, K. H. Maier-Hein,
J. Kleesiek, and R. Stiefelhagen. Detailed annotations of chest x-rays via et projection for report understanding. In Proceedings of the 33th British Machine Vision Conference (BMVC), 2022.
[104] A. Serag, P. Aljabar, G. Ball, S. Counsell, J. P. Boardman, M. A. Rutherford, A. D. Edwards, J. V. Hajnal, and D. Rueckert. Construction of a consistent high-definition spatio-temporal atlas of the developing brain using adaptive kernel regression. Neuroimage, 59(3):2255-222) 2012.
16

---

[105] A. A. A. Setio, A. Traverso, T. De Bel, M. S. Berens, C. Van Den Bogaard, P. Cerello. H. Chen, Q. Dou, M. E. Fantacci. Geurts, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the lunal6 challenge. Medical image analysis, 42:1-13, 2017.
[106] A. L.. Simpson, M. Antonelli, S. Bakas, M. Bilello, K. Farahani, B. Van Ginneken, A. Kopp- Schneider, B. A. Landman, G. Litjens, B. Menze, et al. A large annotated medical im- age dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv:1902. 09063, 2019.
[107] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pages 2256-2263 PMLR, 2015.
[108] Y. Song, Zheng, L.. Lei, Z. Ni, B. Zhao, and Y. Hu. CT2US: Cross-modal transfer learning for kidney segmentation in ultrasound images with synthesized data. Ultrasonics, 122:106706, 2022.
[109] T. Sørensen. A method of establishing groups of equal amplitude in plant sociology based on similarity of species and its application to analyses of the vegetation on danish commons. Biol Skrifter/Kongelige Danske Videnskabernes Selskab., 5:1, 1948.
[110] J. Staal, M. D. Abràmoff, M. Niemeijer, M. A. Viergever, and B. Van Ginneken. Ridge-based vessel segmentation in color images of the retina. JEEE transactions on medical imaging, 23(4):501-509, 2004.
[111] R. Tanno, A. Sacedi, S. Sankaranarayanan, D. C. Alexander, and N. Silberman. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pages 1 11244-11253 2019.
[112] P. V. Tran. A fully convolutional neural network for cardiac segmentation in short-axis mri. arXiv preprint arXiv: 00494, 2016.
[113] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and
I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[114] X. Wang, S. Li, K. Kallidromitis, Y. Kato, K. Kozuka, and T. Darrell. Hierarchical open- vocabulary universal image segmentation. Advances Neural Information Processing Systems, 36. 2024.
[115] X. Wang, W. Wang, Y. Cao, C. Shen, and T. Huang. Images speak in images: A general- painter for m-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8830-6839, 2023.
[116] X. Wang, J. Yang, and T. Darrell. Segment anything without supervision. arXiv preprint arXiv:2406. 20081, 2024.
[117] X. Wang, X. Zhang, Y. Cao, W. Wang, C. Shen, and T. Huang. Seggpt: Segmenting everything in context. arXiv preprint drXiv:2304.0m Xiv:22004.03284, 2023.
[118] J. Wasserthal, H.-C. Breit, M. T. Meyer, M. Pradella, D. Hinck, A. W. Sauter, T. Heye, D. Boll, J. Cyriac, S. Yang, et al. Totalsegmentator: robust segmentation of 104 anatomic structures et images. Radiology: Artificial Intelligence, 5(5), 2023.
[119] W. Wells III, W. Grimson, R. Kikinis, and F. Jolesz. Adaptive segmentation of mri data. IEEE Transactions on Medical Imaging, 15:429-442, 1996.
[120] J. Wolleb, F. Bieder, R. Sandkühler, and P. C. Cattin. Diffusion models for medical anomaly detection. In International Conference on Medical image computing and computer-assisted intervention, pages 35-45. Springer, 2022.
[121] J. Wolleb, R. Sandkühler, F. Bieder. P. Valmaggia, and C. Cattin. Diffusson Models for Implicit Image Segmentation Ensembles. In Medical Imaging with Deep Learning, 2021.
[122] H. E. Wong, J. J. G. Ortiz, J. Guttag, and A. V. Dalca. Multiverseg: Scalable interac- tive segmentation of biomedical imaging datasets with in-context guidance. arXiv preprint arXiv:2412. 15058, 2024.
[123] F. Wong, M. Rakic, J. Guttag, and A. Dalca. Scribbleprompt: Fast and flexible interactive segmentation for any medical image. arXiv preprint arXiv:2312.0733 2023.
17

---

[124] J. Wu and M. Xu. One- pre-propptt to segment all medical images. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11302-11312, 2024.
[125] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for scene understand- ing. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 2018.
[126] L. Zbinden, I.. Doorenbos, T. Pissas, A. T. Huber, R. Sznitman, and P. Márquez-Neila. Stochastic segmentation with conditional categorical diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1119-1129, 2023.
[127] Y. Zhang, M. Xian, H.-D. Cheng, B. Shareef, J. Ding, F. Xu, K. Huang, B. Zhang, C. Ning, and Y. Wang. Busis: A benchmark for breast ultrasound image segmentation. Healthcare, volume 10, page 729. MDPI, 2022.
[128] Y. Zhang, M. Xian, H.-D. Cheng, B. Shareef, J. Ding, F. Xu, K. Huang, B. Zhang, C. Ning, and Y. Wang. Busis: A benchmark for breast ultrasound image segmentation. Healthcare, volume 10, page 729. MDPI, 2022.
[129] Q. Zhao, S. Lyu, W. Bai, L. Cai, B. Liu, M. Wu, X. Sang, M. Yang, and L. Chen. A multi- modality ovarian tumor ultrasound image dataset for unsupervised cross-domain semantic segmentation. CoRR, abs/2207. 2.6799, 2022.
[130] T. Zhao, Y. Gu, J. Yang, N. Usuyama, H. H. Lee, T. Naumann, J. Gao, A. Crabtree, B. Piening, C. Bifulco, et al. Biomedparse: a biomedical foundation model for image parsing of everything everywhere all at once. arXiv preprint arXiv:2405. 12971, 2024.
[131] Z. Zhao, Y. Zhang, C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie. One model to rule them all: Towards universal segmentation for medical images with text prompts. arXiv preprint arXiv:2312.1718 2023.
[132] G. Zheng, C. Chu, D. L. Belavý, B. Ibragimov, R. Korez, T. Vrtovec, H. Hutt, R. Everson, J. Meakin, L. Andrade, al. Evaluation and comparison of 3d intervertebral disc localization and segmentation methods for 3d 12 mr data: A grand challenge. Medical image analysis, 35:327-344, 2017.
[133] X. Zheng, Y. Wang, G. Wang, and J. Liu. Fast and robust segmentation of white blood cell images by self sulf-super learning. Micron, 107:55-71, 2018.
[134] X. Z.uanng. L. Li, C. Payer, D. Štern, M. Urschler, M. P. Heinrich, J. Oster, C. Wang, Ö. Smedby, Bian, et al. Evaluation of algorithms for multi-modality whole heart segmentation: an open-access grand challenge. Medical image analysis, 58: 58:101537, 2019.
[135] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, Wang, L. Wang, J. Gao, and J. Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are justified through rigorous experiments evaluated on held-out datasets both quantitatively and qualitatively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
18

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In addition to the core hypothesis of this work presented in Section we have dedicated limitations section at the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: There are no theoretical results in this paper.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
19

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 3 has a dedicated infrastructure, training, and inference section that specifies parameters and the setup of the model development. The experimental setup is extensively discussed in Section 4 We provide additional information to reproduce the experiments and on model training in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The data we use comes from a collection of public datasets. They are all cited accordingly in the paper but do not have authorization to release the data ourselves. Code will be made available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
20

---

including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
http:///wwlccguieee/eeeeeeeeeeee /oeee soiiooo for more details.
/
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All hyperparameters are specified in the main paper in Section and or in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All error bars are reported the corresponding graphs and for each graph, we explain what they mean. The only graph were error bars are not directly visible is the heat map. We present the error bars in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
21

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: For the infrastructure experiment, we specify the compute resources details in We also provide additional information in the supplemental material.
Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttts/s//euurpp..cccc/bl//eeeee
Answer:
[Yes]
Justification:
We comply to the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have dedicated social impact section that discusses both.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
22

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model a segmentation model, not a pretrained LLM, image generator or scraped dataset. We do caution against the use of our model for out of distribution data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Original owners of assets are credited for code, data and models.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
23

---

Justification: Commented and licensed code will be made public upon acceptance
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No human subjects or crowd-sourcing was involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of L.MMs if is an important, original, or non-standard component of the core methods in this research? Note that if the L.M is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were only used for formatting, editing, and standard code cleaning and development.
24

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:///eaarrrssccoooooooorrrr/e for what should or should not be described.
25

---

A Table of Content
We divide this supplemental material into five major parts:
Frequently asked questions (B). We address some questions that have been asked by colleagues about this work. We think this might be relevant reviewers if this remains unclear in the paper.
Additional analysis on Held-out Datasets (C). We report results for the Set Surface Distance and Set loU metrics. We analyze further architectural choices and their impact on performance. This includes the size of the set and the choice of M and K. We provide per-dataset performance when results are shown aggregated in the main paper.
Additional analysis on Development data (D) We present performance for the Development datasets, which were used to evaluate the generalization capabilities of the model.
Training infrastructure
E We detail our training infrastructure and requirements.
Data F.. We provide additional information on Megamedical and Anatomix as well as the augmen- tations applied.
B Frequently Asked Questions
Instead of Pancakes, could user fine-tune a model on set of curated images? Pancakes tackles the prevalent scenario where there is no preexisting annotated data. It relieves the user of the requirement imposed by existing methods; annotating sufficient images to use for few-shot or in-context methods.
What is the difference between Pancakes and stochastic segmentation methods [0] 5917 Pan-
cakes is focused segmenting images in new tasks, not seen at training, while existing methods are specifically designed to model stochasticity in a given pre-specified task, essentially tackling a different problem. While Pancakes can also capture diversity for fixed single protocol like existing models do, we model the substantially more challenging problem of presenting different plausible (multi-label) protocols for the new task.
What does it mean for segmentations to be consistent across set? If a label ID appears across a set of anatomically similar images, this ID semantically corresponds to the same ROI each image. For example, if label I represents the hippocampus in the first image, it should also represent the hippocampus in the second image.
Why is consistency important? Consistency is critical when the user wants to analyze the same set of structures for multiple images is prevalent population and longitudinal studies.
Is it possible to achieve consistency through post-procsssing if method is inconsistent? For previous methods, matching segmentation maps across scans is not trivial for several reasons. Among them: (1) The segmentation maps in different scans often different that there labels to match. The same segmentation model could segment images on different bases and different images contain different structures (2) Even when the same anatomical structure is segmented in two different images, they usually don't 't share the same label ID or don't 't have the same size as shown in Figure? 2 This makes automated matching as post-processing step ill-posed.
clear
How are the sets treated at training? We train with input tensors of dimension B x S x C' x H W, for batch B, set S, channel C, image height H and width W. We flatten these tensors to (B x S) x C' x H x W so that we can use architectures based on 2D-convolutions and 2D-UNets.
What is the intuition behind the min Dice loss? Using a regular expectation over the candidate label maps would lead the model to regress to the mean. This might be particularly harmful when there is high uncertainty around the set of plausible labels to output. Instead, using the best values leads to more diverse predictions. We refer to [96] for more extensive comment on this type of loss.
26

---

WBC
HipxRay
Qubig
Pancakes
SAM
SP-SAM
UnSAM
Med8 AM
Figure 11:
Pancakes and baselines set consistency. Evaluated on held-out datasets (left to right: WBC, HipXRAY, QUBIQ Prostate), Pancakes the method that is the most consistent across predictions compared to the baselines.
C Additional Analysis on Held-out Datasets
We present more in-depth results for the held-out datasets that remained unseen until the final phase of evaluation.
C.1 Qualitative comparison with baselines
We visually compare predictions for three held out datasets: WBC, HipXRay and QubiqProstate. Figure IT shows that even though some baselines produce more accurate individual labels, Pancakes provides the most consistent labels across images of the same set.
C.2 Interpolation
This additional visualization explores the space of protocols M for multiple numbers of labels: K. For M = 16, we sample the 16 protocols available for different K values: [5, 10, 15, 20, 25). We show in Figure 12 how this impacts the produced label maps for two samples in the WBC dataset. We find that segmentation maps that come from protocols close to one another in the embedding space similar. Interestingly, the label space interpolated relatively smooth, indicated by the color and shape variations.
C.3 Additional Metrics
report results evaluated using two additional metrics: Set IoU and Set surface distance. They are computed in the same way as Ser Dice but replacing Dice score with loU and Surface Distance respectively. For the surface distance, keep the percentile 95, When the set size one (S = 1), the set metric reduces to the standard metric, ignoring consistency.
In Figure 13
C.4 Per set size performance
We report additional per-set-size results that were summarized in the main paper.
27

---

K = 5
10
K = 15
K 20
25
==5
K - 10
= 15
20
K = 25
Figure 12: Interpolation Analysis. By covering the space of protocol (rows) for several fixed K values, we observe that label maps resulting from protocols close to another the embedding space similar.
M-8, K=
Surface Dieeance
40
35
MedSAV LESAM
20
15
SAM
Tarcakes
Set Size
Seet Size
Figure 13: Evaluation additional metries: loU and surface distance. For individual predictions (S = 1), Pancakes is comparable to SAM and outperforms baselines. As the set size S increases, Pancakes is the only model whose performance is not severely degraded, as it can produce consistent segmentations.
28

---

90
90
60
40
40
30
20
BURD
D0mT
Hopoaa
QublgTei
QubbeTT2
sco
STARE
WBC
UDC
BUD
Hipanaj
OsbigTRI
QubigT122
STARE
LDC
(a) S
2
90
90
70
70
50
20
20
BUND
HipaRay
QublgTel
QubetTE2
sco
STARE
WBCc
BuRa
Don
repaaay
QuileTe1
UubiyTRc
sco
WBC
uDc
S
(d) S
Figure 14: Per-dataset Set Dice performance under various set sizes. We evaluate Pancakes and
baselines with various set sizes: 1, 2, and 5, For Pancakes, evaluate with M 8 and K 20.
Per-dataset Set Dice performance. This experiment evaluates how segmentation accuracy and label consistency vary jointly with set size. Figure [4 shows that Pancakes's ' improvement over the baselines increases with the size. Figure 14a focuses accuracy only the size one (S = 1). In this case, Pancakes comparable to the baselines (Figure 14a We observe a large variability across different datasets.
Influence of M and K. Figure 15 shows the influence of the number of protocols and labels on the Pancakes predictions as the set size varies. As the number of protocols increases, performance increases. Best performing K seems to be in the middle of the range for K=20. Results are consistent across set sizes.
D Additional Analysis on Development Datasets
We include performance on the Development datasets that were used to evaluate generalization capabilities during model development: ACDC [1]. PanDental
2 and SpineWeb [132).
We evaluate on the test split of all datasets. When reporting performance with certain set size, we exclude tasks with subjects fewer than this set size. (For example, the test split of Web has only 2 subjects, so we do not include it in set size and experiments.) We report Pancakes performance with M = 8 and K 20. Figure 16 shows Pancakes demonstrates both accurate and consistent predictions compared to the baselines.
Per-Dataset Set Dice Performance for Different Set Sizes. Figure 17 shows separated by set size.
Influence of M and K. Figure 18 shows effect of number of protocols and labels per protocol prediction accuracy, as set size varies. The trend aligns with results using the held-out datasets (
Learning a complete protocol. The primary objective of Pancakes is to propose diverse set of consistent protocols for an image group. We do not enforce constraints across protocols produced, but at times it would be good to do so. For example, symmetric labels (left and right ventricles or posterior and anterior hippocampus) in the same protocol may desirable. One way to incorporate this into the Pancakes framework is to apply the loss function to two randomly chosen segmentation
29

---

51.99
55.96
58.49
60.05
59.13
48.76
52.12
53.98
54.19
53.82
2
-
-
2
=
4.14
3.91
4.31
4.35
4.06
4.23
3.95
4.40
4.48
4.37
58.33
61.16
62.71
63.72
63.43
55.75
55.49
57.24
58.31
58.56
65
4
3.96
4.04
4.22
4.28
4.15
4.09
3.87
4.20
4.29
4.41
M
62.45
64.37
68.73
67.58
67.38
58.92
59.31
64.18
59.92
60.40
8
4.09
4.13
4.36
4.19
4.20
4.32
4.00
4.35
4.05
4.33
-55
55
64.20
66 .40
70.69
70.01
68.97
60.29
60.01
65.50
63.85
61.86
16
=:
16
3.96
4.06
4.33
4.31
4.16
4.09
3.76
4.45
4.28
4.32
10
30
40
10
20
30
K
K
(a) S
(b) =
49.68
52.75
53.27
53.80
53.60
53.46
57.11
55.95
56.46
55.97
2
70
70
3.82
3.58
3.72
3.77
3.76
3.72
3.97
3.60
3.75
3.82
55.88
55 35.66
58.38
57.20
58.19
58.59
59.43
60.00
61.33
60.90
4
+
=
3.78
3.48
3.75
3.69
3.80
3.71
3.53
3.54
3.71
3.70
M
60
M
60
58.40
58.54
64.55
60.22
60.41
60.78
62.03
66.63
61.03
63.24
+
+
+:
4.11
3.78
4.16
3.58
3.73
4.07
3.59
3.76
2.96
3.82
59.44
59.24
65.47
63.81
61.96
61.35
62.01
67.58
66.25
64.00
16
16
3.94
3.57
4.20
3.95
3.78
50
3.83
3.31
3.71
3.46
3.53
50
5
10
20
30
40
5
10
20 K
30
40
(c) S
(d) S =
Figure 15: Influence of varying M and K for various set sizes on the Held-out datasets. We evaluate Pancakes with various sizes, number of protocols (M) and number of labels in each protocol (K).
targets in multi-label dataset, rather than As preliminary experiment, experiment on the OASIS brain dataset, and found it effective at learning a protocol as shown in Figure 19, We planning explore this more in future work.
E
Training Infrastructure
Our model trained using 45G of memory single node of an NVIDIA DGX A100 machine using two cores. We use batch size of and the Adam optimizer with a learning rate of 0.0001. We use PReLU activations and convolution layers with 32 features, kernel size 3 and stride 1.
F
Data
F.1
Megamedical
The images are 2D and resized to 128 x 128. Complete tables of the datasets, split by data subgroup (Training. Development, Held-oul) are shown Tables 4 6 Megamedical covers wide range of modalities (MRI, ultra-sound. XRay) and anatomies (organs, bones, substructures and fine structures like vessels).
train our main experiment on Megamedical 16 96 123
30

---

60
50
Set Dice
40
30
MedSAM
20
UnSAM
ScribblePrompt
10
SAM
Pancakes
1
2
3
5
Set Size
Figure 16: Pancakes is both consistent and accurate compared to baselines. We evaluate Pancakes and the baselines on Development datasets: ACDC, PanDental and SpineWeb. When evaluated solely on accuracy (S 1). Pancakes is comparable SAM and outperforms the other baselines. As S increases, Pancakes is the only model whose performance is not degraded.
30
20
10
10
SpineWeb
1
S=
60
50
50
40
40
30
ACDC
PanDental
ACDC
PanDental
(c) S
(d) 5
Figure 17: Per-dataset Set Dice performance under various set sizes for the Development datasets. We evaluate Pancakes and baselines on development datasets (ACDC, PanDental and SpineWeb) Web) with various sizes: For Pancakes, we use M = and K 20.
and 5. If dataset has fewer subjects than the size, exclude
F.2 Anatomix
and specifically on Anatomix 27 we use synthetic data to complement Megamedical and limit overfitting. To generate Anatomix label maps, we sample random labels from the TotalSegmentator dataset [118]. We sample between 20 and 40 labels and generate 128 x 128 128 label map. Once this 3D label map is generated, we randomly sample an axis and then slice between slice ID 25 and 100. With probability 50% we will split labels. In that case, labels are reassigned so that given label can only be composed of contiguous pixels. We also assign to background (label ID 0) any label whose size is smaller than 20 pixels. This gives us the label map template for given set, from which we are going to generate the images and label maps for each element in the set. We generate the images by randomly assigning intensity value to each label. We then apply a series of augmentations that are independent for each element in the set. These augmentations include: Gaussian blur, Gaussian noise, Perlin noise, elastic and affine transforms,
Building on 13 16
3D
31

---

42.63
45.06
50.82
47.87
47.04
38.95
41.91
46.63
42.39
41.87
2
+
=
2
8.52
7.90
8.45
7.88
8.00
7.85
7.56
7.57
6.58
6.62
48.02
50.25
54.81
53.37
51.13
65
43.76
46.05
50.05
47.61
45.15
65
8.00
8.93
9.44
8.94
8.81
8.22
8.77
8.51
7.44
7.27
M
51.88
56.12
59.03
58.04
56.03
47.84
50.48
52.61
51.63
49.08
8
= 9.48
10.09
10.12
9.54
9.86
10.30
9.28
8.63
9.34
8.25
-55
55
54.33
59.15
61.67
60.53
58.90
49.72
53.31
55.69
54.16
52.15
16
-
+:
=: 10.03
16
-:
9.99
9.82
9.68
9.70
10.42
9.21
9.14
9.45
8.94
50
5
10
20
30
40
5
10
20
30
40
K
(a) S
(b)
41.82
43.46
47.72
42.44
41.53
41.64
42.55
46.78
40.67
39.71
=
= 7.23
2
= 7.12
7.76
7.99
7.83
7.10
7.62
7.94
7.92
7.43
45.37
48.13
51.22
48.79
44.91
45.01
47.13
49.60
46.
42.53
65
+
+
= 7.76
=
9.29
9.33
8.35
7.70
9.21
9.26
8.64
7.82
7.75
M
M
50.70
52.96
53.42
52.22
49.00
50.13
51.58
51.53
50.16
46.95
8
10.93
9.47
8.45
9.97
8.51
10.91
9.96
8.69
10.25
8.37
-55
55
51.98
55.41
57.51
55.75
52.25
51.20
53.82
55.31
53.06
49.44
16
=
16
=
11.04
9.32
8.
9.49
9.00
10.99
9.58
9.05
9.89
8.87
10
30
40
5
10
20
30
K
(c) S
(d) S
Figure 18: Influence of varying M and K in various set sizes for the Development datasets. On development datasets, evaluate Pancakes with various sizes, number of protocols (M) and number of labels in each protocol (K).
Img
Protocal 3, Img 0
Protocol 1, Img 1
Protocol 1, Img 2
Protocol 2, Pred
Protocal Pred 1
Protocal 2, Pred
Protocal
Protocol 2,
Protocol
Protocol Pred
Protocol Pred
Protocal Pred
Available Prococok
Par cales prodictions
Figure 19: Learning full protocol. Training OASIS, learn accurately complete protocol with our loss modification.
32

---

Set
Set
Set 3
Images
Maps
Figure 20: Anatomix Examples. Label maps and images within a set are very similar to one another, sharing similar relative locations for each structure. The main difference lies in the augmentations applied, to the images individually (Gaussian noise for example), but also to both the images and label maps (elastic deformation for example).
and contrast variations. To sample binary ground truth label at training, we sample a label that is present in each image of the Example sets generated are shown in Figure 20
F.3
Data Augmentation
At training, we apply a series of augmentations to our set to improve generalization capabilities. We types of augmentations, within-sst augmentation applied independently
distinguish between to each element in the set - and across-sel augmentation - applied consistently to each element in the set. For each augmentation sampled, the corresponding parameters are sampled uniformly from a pre-defined range. Table showws the list of all augmentations applied, the probability of each augmentation per iteration and the parameter ranges we sample from.
33

---

Table 3: Augmentations used train We apply augmentations either independently within a set (Top) to all the elements of set (Bottom). We sample cach parameter from the uniform distribution (LI) within the ranges defined in the Parameters column.
(a) Within-Set Augmentation
Augmentations
p
Parameters
degrees ~ 27(2272555 25)
Random Affine
0.25
translate~ (.10
scale~ 1(00.91.1)
brightness~ (((001100.)
Brightness Contrast
0.5 contrast ~ 1.(00.5,
at U(1,2.5)
Elastic Transform
0.8
((7
Sharpness
0.25
sharpness= 3
Flip Intensities
0.5
None
1))
Gaussian Blur
0.25
k=5
~~ ((O, 0.05)
Gaussian Noise
0.25
0 (10, 0.05)
(b) Across-Set Augmentation
Augmentations
Parameters
degrees W(0 360)
Random Affine
0.5 scale~ 4(0.8, 1.1)
translate~ 0.2)
brightness~ 0((-0.1,0.1)
Brightness Contrast
0.5 contrast~ 4(0088,11 1.2)
4(0.1,
Gaussian Blur
0.5
k: = 5
1 0.00)
Gaussian Noise
0.5
0 (.005)
((1,2)
Elastic Transform
0.5
~~ U(6,8) 8)
Sharpness
0.5
sharpness
Horizontal Flip
0.5
None
Vertical Flip
0.5
None
34

---

Table 4: Collection of datasets in Megamedical used for training, The entry # of scans is the number of unique (subject, modality) pairs for each dataset.
Dataset Name
Description
#
of
Image Modalities
Scans
AMOS
49
Abdominal organ segmentation
240
CT. MRI
BBBC003
72
Mouse embryos
15
Microscopy
BBBC038
Nuclei images
670
Microscopy
BrainDev.
33
34
63
Adult and Neonatal Brain Atlases
53
multi- molti-mmdall MRI
104
BRATS 6
Brain tumors
6,096
multi-modaa MRI
BTCV 665
Abdominal Organs
CT
BUS 128
Breast tumor
163
Ultrasound
CAMUS 66
Four-chamber and Apical two-
500
Ultrasound
chamber heart
CDemris
[5
Human Left Atrial Wall
CMR
CHAOS
53
55
Abdominal organs (liver, kidneys,
40
CT,
T2-weighted
spleen)
MRI
CheXplanation [T-ORG[98]
101
Chest X-Ray observations
170
X-Ray
Abdominal organ segmentation (over-
140
CT
lap with LiTS)
DRIVE 110
Blood vessels in retinal images
20
Optical camera
ROphtha 25
Eye Microaneurysms and Diabetic
102
Optical camera
Retinopathy
FeTA 889
Fetal brain structures
80
Fetal MRI Fetoscopic optical camera Ultrasound
FetoPlac 9
Placenta vessel
6
HMC-QU 26
57
4-chamber (A4C) and apical 2- chamber (A2C) left wall
292
12CVB 67
Prostate (peripheral zone, central 19
T2-weighted MRI
gland)
IDRID 93
Diabetic Retinopathy
54
Optical camera
ISLES 42
Ischemic stroke lesion
180
multi-modal MRI
KiTS
Kidney and kidney tumor
210
CT
LGGFlair
TCIA lower-grade glioma brain tumor
110
MRI
LiTS 12
Liver Tumor
131
CT
LUNA 105
Lungs
888
CT
MCIC 32
Multi-site Brain
regions
of
390
T1-weighted MRI
Schizophrenic patients
MSD [166
Collection of 10 Medical Segmenta-
3,225
CT.
multi-modal
tion Datasets
MRI
NCI-ISBI 14
Prostate
30
T2-weighted MRI
OASIS [45 80
Brain anatomy
414
T1-weighted MRI
OCTA500
(69
Retinal vascular
500
OCT/OCTA
PAXRay [13
Thoracic organs
880
X-Ray
PROMISE12
Prostate
37
T2-weighted MRI
PPMI 81
Brain regions of Parkinson patients
1,130
T1-weighted MRI
QUBIQ 83
Brain, kidney, pancreas
209
MRI TI, Mul-
timodal
MRI.
CT
ROSE 778
Retinal vessel
117
OCT/OCTA
SegTHOO
Thoracic organs (heart, trachea, esoph-
CT
agus)
ToothSeg 47
Individual teeth
598
X-Ray
WMH 62
White matter hyper-intensities
60
multi-modal MRI
WORD
Organ segmentation
120
CT
35

---

Table 5: Development datasets. Datasets used to evaluate the generalization capabilities of model and model development.
Dataset Name
Description
of
Image Modalities
Scans
ACDC [
Left and right ventricular endocardium
99
cine-MRI
PanDental
Mandible and Teeth Vertebrae
215
X-Ray
Spine Web 32
15
T2-weighted MRI
Table 6: Held-out datasets. Datasets that remained unseen until the final evaluation phase.
Dataset Name
Description
of
Image Modalities
Scans
BUID
Breast tumors
647
Ultrasound
DDTI
Thyroid
472
Ultrasound
HipXRay 37
Ilium and femur
140
X-Ray
QUBIQ 83
Prostate
209
MRI TI, Mul-
timodal
MRI,
CT
SCD 94
Sunnybrook Cardiac Multi-Dataset
100
cine-MRI
Collection
LIDC-IDRI 55
Lung Nodules
1,018
CT
STARE
Blood vessels in retinal images
Optical camera
WBC 333
White blood cell and nucleus
400
Microscopy
36


Reviews:
[{"review_id": "MFDhYUlIFw", "reviewer": "Reviewer_kGwA", "summary": "There are many ways to segment a given biomedical image: Segmenting regions, vessels, detecting tumors etc. Most models are trained on only a single segmentation task, or require prompting and can be inconsistent between samples. The authors introduce \"pancakes\", a model and a training scheme that learns to segment up to K classes on M different tasks at the same time. Rather than being preset in advance, these segmentation modalities are learned automatically from training data by attempting to learn a segmentation classes that generalize and remain consistent between many datasets and samples. The approach is evaluated on a variety of datasets and input modalities (CT, MRI etc.) and compared with other universal models such as MedSAM and UnSAM, which it outperforms.", "questions": "I address my questions in the weaknesses section. Right now, while I appreciate the task and contribution, the many details that are unclear me to prevent me from recommending acceptance with a clear conscience. If these become clear after the rebuttal, I would gladly reassess my score. I thank the authors in advance for their time.", "limitations": "The authors address limitations in their paper. I do not believe that there are further concerns about ethics or societal impact.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n* The task of discovering and reproducing labels consistent across many datasets and segmentation tasks rather than training for a handful of preset labels is an interesting one. \n* The core parts of the approach are explained clearly, and the metrics for evaluating label consistency across tasks (Set Dice) make sense. \n* The training and evaluation is solid, and comparisons to more \"universal\" models that could be used for the same task are favorable. \n\nWeaknesses\n\n* Some parts of the training are unclear to me. According to line 130-132, all protocols and labels for a sampled M, K are predicted at every training iteration. Then, according to Eq. 7, at each iteration, only the protocol-label pair that has the best prediction is optimized to encourage consistency. How is label leak across protocols prevented? Example: Suppose a dataset has Anterior and Posterior segmentations of the hippocampus as labels. What prevents the Anterior label from ending up in Protocol 0 and Posterior in Protocol 1? How do we make sure related tasks end up in the same protocol? As far as I understand, each labeling task ends up in a random protocol/label slot, which I am sure must be a misunderstanding. \n* The use case for such a model is also a bit unclear. When running Pancakes on a new dataset for exploratory purposes, how can a user know which labels are what, and which protocols make sense? It is possible that some of the learned protocols make no sense on a given dataset, and in this case the \"prompting\" of a more universal model towards a specific even sounds like an advantage rather than a disadvantage. How would seeing a number of segmentations on a set of protocols and labels that are not well defined be helpful? I think it would be nice to clarify this with some clear real-world scenarios rather than only testing on a scenario like \"S = 3, M = 8 and K = 20 sounds good\" which seems intuitive but not based on a concrete use case.  \n* Evaluation against single-protocol models is unavailable. While the universality of the approach does indeed call for comparison to universal methods, it seems straightforward to expect that Pancakes would outperform them since they are not trained for consistency. It would be interesting to know how well the model performs against run-of-the-mill segmentation models. For example, if I want to focus on three specific protocols, what would be the advantage of using Pancakes rather than 3 separate models trained specifically on these protocols? Is the generality worth the performance trade-off? Or, does the generality and multi-model training actually improve performance? This is valuable knowledge. \n* Fast inference time is put forward as an advantage of Pancakes, having only 0.22M parameters. This comes as a surprise in our current era of larger and larger models. Since this is put forward as an advantage, I think it worth exploring. Why was a small model chosen specifically? What happens with a number of parameters comparable to other models? It seems surprising that learning across large amounts of data can be this successful with such a small number of parameters at a glance. \n* While the task introduced and being evaluated on is novel, most of the technical components are not. As an example, the loss being optimized on the best predicted mask at every existing iteration is done in SAM (this is also stated in the paper). \n* The Limitations section states that Pancakes is strong in a resource-constrained setting, but intuitively it seems like it would do better with a larger number of datasets due to the consistency enforced between them. What is the argument for this?", "was_revised": true, "final_justification": "I appreciated the authors' responses, but was still not convinced about some of the points as stated in my comments to the authors. In the end, I raised my score from 2 to 3.", "rebuttal": "We thank the reviewer for their thorough and useful feedback. We are glad they found the task novel, and appreciated our training and evaluation framework and Pancake’s strong performance. We are also pleased that they found the paper interesting and clear. Below, we address the remaining questions on the protocol use, method applications, baseline selection, and the evaluation setting.\n\n**Could labels leak across protocols?**\n\nThe primary objective of Pancakes is to propose a diverse set of consistent protocols for an image group. We do not enforce constraints across protocols produced, but we agree that at times it would be good to do so. For example, symmetric labels (left and right ventricles or posterior and anterior hippocampus) in the same protocol may be desirable. One way to  incorporate this into the Pancakes framework is to apply the loss function to two randomly chosen segmentation targets in a multi-label dataset, rather than one. To validate this, we ran an experiment on the OASIS brain dataset (2 protocols with 5 and 25 labels respectively) during the rebuttal period, and found it effective. We thank the reviewer for raising this important point and will include the method and results in the revised manuscript. \n\n**Use case clarification.** \n\nWe will clarify that Pancakes is not intended to replace existing clinically validated segmentation protocols. If there is an existing tool for a particular protocol, we would advise using it. Rather, our goal is to support a new capability: enabling biomedical researchers and clinicians to explore a diverse set of plausible, semantically consistent segmentations for an unseen collection of images. After Pancakes has generated the label maps, a researcher or clinician selects which of the proposed protocols best aligns with their intended downstream use (e.g., anatomical volume analysis). We envision two broad classes of use:\n\n(1) *Rapidly segmenting new protocols*. New protocols are frequently introduced [133, 136, 137], and there is a need to produce corresponding segmentations. If a scientist has a particular protocol in mind, but there are no existing tools for segmenting it, they can choose the protocol from Pancakes that best aligns with their intended use. This choice will often suffice. In cases where Pancakes’ segmentations are not sufficiently accurate–for example, when targets are more out-of-distribution than those seen in training–the segmentation maps can offer excellent initializations to interactive segmentation systems such as ScribblePrompt [120]. To demonstrate this, we performed a preliminary experiment and found that:\n\n- On average, Pancakes’ predictions can be improved by 5 Dice points with a *single interactive click*, for set sizes larger than one.\n\n- On average, using ScribblePrompt with a Pancakes-initialized segmentation reduces the number of required interactions by half. If used alone, it takes ScribblePrompt 5 to 8 clicks to reach optimal quality predictions, while, when initialized with Pancakes’ predictions, ScribblePrompt can reach them in 3 to 4 clicks.\n\nSince  figures are not permitted in the rebuttal, we cannot provide plots for this experiment, but will include them in the manuscript.\n\n(2)  *Exploratory population analysis*. Pancakes will support users in discovering or selecting segmentation strategies appropriate for their scientific or clinical questions. For example, a clinical scientist who studies how anatomy relates to some outcome (e.g., progression of a disease) or predictor (e.g., genetics) can use Pancakes to quickly extract segmentations of multiple candidate anatomical regions that have never been segmented, compute their volumes, and test correlations with clinical outcomes thereby identifying promising candidate regions.\n\n**How can a user identify structures and know which protocols make sense?** \n\nPancakes is intended for biomedical experts. We assume that as they use the tool, they can identify the mapping between a label and a known structure. Because Pancakes provides consistent segmentations for groups of images, this provides some context on what labels represent. In future work, we plan to investigate automating the mapping between a structure and known protocols. \n\n\n**What would be the advantage of using Pancakes rather than 3 separate models trained specifically on these protocols?** \n\nPancakes is not intended to replace clinically validated segmentation protocols. If the user knows which targets they wish to segment, and good models exist for those targets, those models should be used. Pancakes is meant for scenarios where the targets are unknown or for which there are no existing segmented scans or segmentation models. \n\n**While the task introduced and being evaluated on is novel, most of the technical components are not. As an example, the loss being optimized on the best predicted mask at every existing iteration is done in SAM (this is also stated in the paper).** \n\nWe agree that the individual components of our framework have been explored in prior contexts. Our key contribution is in formulating an interesting new problem, and in carefully adapting and integrating techniques to solve it. This includes developing a version of the loss function that supports consistency, adapting the synthetic data so that the produced samples are consistent and proposing a lightweight architecture that can efficiently sample protocols. However, we emphasize that our main contribution lies in introducing a fundamentally new framing to medical image segmentation, and alleviating the annotation requirements on the user.\n\n**Small model and resource constrained setting** \n\nThere is strong evidence that UNet-based architectures achieve performance similar to  transformer-based architectures in medical image segmentation [134, 135]. Using a relatively small number of parameters limits overfitting, and especially task-overfitting, where the model remembers the structures to segment instead of proposing diverse segmentations. An additional benefit of using a small model is inference time efficiency. We did not have the time to fully train larger models during the rebuttal period. However, preliminary results indicate that increasing the number of parameters does not lead to an improvement in performance. We also highlight that our proposed framework is not dependent on the choice of the neural network that captures the function $f_{\\theta_f}(.)$, which infers the distribution parameters $\\phi$. The UNet implementation of $f_{\\theta_f}$ could be replaced with another network, such as the SAM featurizer. We will discuss this in the revised manuscript.\n\n[133]: Merdietio Boedi, Rizky, et al. \"*3D segmentation of dental crown for volumetric age estimation with CBCT imaging.*\" International Journal of Legal Medicine 137.1 (2023): 123-130. \n\n[134]: Isensee, F. et al. (2024). *nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation*.  In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15009. Springer, Cham.   \n\n[135]: Carsen Stringer and Marius Pachitariu. *Transformers do not outperform cellpose*. bioRxiv, pp. 2024–04, 2024 \n\n[136]: *TopBrain 2025 Grand Challenge* \n\n[137]: Greve, Douglas N., et al. \"*A deep learning toolbox for automatic segmentation of subcortical limbic structures from MRI images.*\" Neuroimage 244 (2021): 118610."}, {"review_id": "6puuoV9R0J", "reviewer": "Reviewer_1J1k", "summary": "This paper addresses an important yet under-explored area in medical imaging, particularly where foundation models are concerned, that being the lack of automated segmentation models capable of offering segmentation predictions across multiple segmentation protocols. The authors propose a method, dubbed Pancakes, that takes an image as input and outputs a distribution over segmentation protocols. Their proposed segmentation sampling mechanism then produces several complete, semantically-consistent, multi-label segmentation maps from diverse protocols for that image.", "questions": "* In Figure 2, what are the three methods? Are they automatic segmentation foundation models (if so, which ones?) or are they the three protocols highlighted in Figure 1? \n\n* Did the authors evaluate the impact that the inclusion of synthetic data had on their proposed method? It would be interesting to see how stark the performance drop off is if Pancakes had to rely solely on the real medical imaging data. \n\n* As shown in Figure 5, in the majority of cases, the second-best model is SAM. This is interesting, as SAM is not pre-trained specifically on medical images, yet as shown in the figure, SAM often outperforms the baselines that were. What is the authors’ hypothesis for this? \n\n* On page 8, the authors state, “For our main evaluation, we use Set Dice of S = 3, and use M = 8 and K = 20 for Pancakes. The values were chosen based on our guess for typical values that would be used by a biomedical researcher who wishes to get a general and coherent understand of a set of related images”. Can the authors please elaborate on why these particular values were chosen as the “guess”? It is interesting that these same values demonstrate the best performance as shown in Figure 9. So, were these values truly a guess, or did the results shown in Figure 9 influence the selection of these values? \n\n* In section 6, the authors highlight that in their work, they assumed “The user has an idea of the labels to segment and can select a priori a few K values”. Could the authors elaborate how this requirement of the user compares to the requirement of a user to provide a hierarchy of protocols to existing multi-protocol segmentation models as discussed in the Related Work section?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n* The approach is well-motivated, as automated multi-protocol segmentation models are woefully underexplored in the literature. \n* The authors’ proposed approach seems fairly simple and easy-to-understand, meaning that replication should be fairly straightforward. \n* The proposed Pancakes model demonstrates strong performance against relevant automated segmentation foundation models, in both accuracy and semantic consistency. \n* This improvement in performance also comes with a significant reduction in the number of parameters and inference time, making Pancakes much more efficient compared to existing methods. \n\n\n$~$\n\nWeaknesses: \n\n* The performance evaluation of Pancakes seems to be lacking in robustness for the scope of the claims the authors are making. In Figures 5 and 8, Pancakes is compared against just four segmentation baselines, of which only two are pre-trained using specifically medical data. In their Related Work section, the authors discuss universal segmentation models (notably [45, 55, 98]) and several existing multi-protocol segmentation methods [19, 73, 96, 107], yet they do not compare quantitatively assess the performance of Pancakes against any of these models. Granted, the task the authors’ are exploring appears novel, but more elaboration as to why these existing universal and multi-protocol segmentation methods couldn’t be adapted in a manner where Pancakes could be compared against them. \n* Other than qualitative assessment, the authors only evaluate the performance of Pancake on one quantitative metric: their novel Set Dice. While the authors do highlight the limitations of the traditional Dice metric for their specific task, relying on just a single quantitative metric of segmentation accuracy leaves questions regarding whether the strong performance of Pancakes is truly robust against various measures of segmentation quality. More discussion on the limitations of other traditional metrics for this task would have been appreciated, and the evaluation would have been enhanced by considering at least one more quantitative metric. \n* While the performance improvements offered by Pancakes are sufficient enough for this not to be a major problem, it would have perhaps been better to represent the results shown in Figure 5 as a table as opposed to a bar chart, as distinguishing performance can be tricky, particularly if two models perform similarly to each other. \n* Some figures that would have enhanced the comparison of Pancakes to the baseline models, such as a figure showing a qualitative segmentation comparison between them, are missing.", "was_revised": true, "final_justification": "I thank the authors for effectively addressing most of the comments. I am happy to raise my rating from 4 to 5. I suggest the authors to incorporate the discussion points and clarifications into the main text.", "rebuttal": "We thank the reviewer for their thorough and useful feedback. We are glad they found the task novel, and appreciated our training and evaluation framework and Pancakes’ strong performance. We are also pleased that they found the paper well-motivated and clear. Below, we address the reviewers questions about  the protocols M and candidates K, the choice of baselines, and the metric used.\n\n**Limitations of traditional metrics.** \n\nA distinguishing attribute of our metric is that it takes into account the consistency of the labels produced across images of the same set. This aspect is not taken into account by other metrics. This means that the same structure (for example, a ventricle) could be classified satisfactorily, but be assigned different labels. Because structures are very rarely aligned, identifying the matching labels across different images is a challenge that is ignored with the common metrics. We will clarify this point in the paper.\n\n**Additional quantitative metrics.** \n\n- Note that the value for set size S=1 in Figure 8 corresponds to the traditional Dice score metric. \n\n- As the reviewer suggests, we included additional metrics. In addition to Set Dice, we now include Set Intersection over Union (IoU) and Set Surface Distance (percentile 95%), \n\n\n_Results_ \n\nWe find that for Set Surface Distance, Pancakes outperforms all the baselines. For Set IoU, Pancakes performs similarly to SAM for S=1 (looking at a single image, equivalent to the traditional IoU metric) and outperforms the baselines in all the other cases.\n\n| Surface Distance          | Set Size = 1     | Set Size = 2     | Set Size = 3     | Set Size = 5     |\n|----------------|------------------|------------------|------------------|------------------|\n| MedSAM         | $16.5 \\pm 2.0$   | $24.2 \\pm 2.5$   | $30.3 \\pm 2.3$   | $36.7 \\pm 1.9$   |\n| ScribblePrompt | $21.5 \\pm 8.5$   | $26.9 \\pm 6.2$   | $31.9 \\pm 6.0$   | $34.4 \\pm 4.1$   |\n| SAM            | $15.9 \\pm 6.5$   | $26.9 \\pm 3.2$   | $32.6 \\pm 2.3$   | $37.6 \\pm 1.9$   |\n| Pancakes       | $8.2 \\pm 1.3$    | $10.6 \\pm 1.8$   | $12.0 \\pm 2.1$   | $13.0 \\pm 2.5$   |\n\n| IoU          | Set Size = 1     | Set Size = 2     | Set Size = 3     | Set Size = 5     |\n|----------------|------------------|------------------|------------------|------------------|\n| MedSAM         | $15.7 \\pm 5.5$   | $11.4 \\pm 4.0$   | $9.9 \\pm 3.7$    | $8.3 \\pm 3.5$    |\n| ScribblePrompt | $43.2 \\pm 7.7$   | $30.1 \\pm 5.1$   | $27.1 \\pm 4.4$   | $23.4 \\pm 3.4$   |\n| SAM            | $52.0 \\pm 7.0$   | $33.9 \\pm 4.0$   | $27.5 \\pm 3.0$   | $22.1 \\pm 2.7$   |\n| Pancakes       | $51.9 \\pm 6.1$   | $47.8 \\pm 6.5$   | $48.8 \\pm 6.2$   | $49.7 \\pm 5.8$   |\n\n**Qualitative visualizations for the baselines.**\n\nSince figures are not permitted in the rebuttal, we cannot provide additional qualitative visualizations. Figure 2 shows examples of the segmentations produced by the baselines (left to right, SAM with 2 different hyperparameter choices and UnSAM). The labels produced are plausible but unlike the Pancakes labels, these labels are not consistent across images. Moreover, they are not organized, many of them overlap or some pixels are left without a label assignment. These findings are consistent across datasets and we will include them in the manuscript as well as the corresponding visualizations.\n\n**In Figure 2, what are the three methods?**\n\nThese are all automatic segmentation foundation models. Left to right, we include SAM for different hyperparameter values in automatic mode and unSAM in automatic mode. In our evaluation, we select different hyperparameter values in an attempt to collect as many valid masks as possible. \n\n**Impact of the synthetic data.**\n\nWe will add  an ablation study comparing training Pancakes with real data only, synthetic data only, or both. When trained with both real and synthetic data, Pancakes consistently outperforms the other variants in Dice score, (p<0.05, paired Student-t test), for M=16:\n| Data        | Set Size = 1     | Set Size = 2     | Set Size = 3     | Set Size = 5     |\n|--------------|------------------|------------------|------------------|------------------|\n| Both         | $73.2 \\pm 5.5$   | $67.3 \\pm 6.0$   | $67.4 \\pm 6.0$   | $68.4 \\pm 5.6$   |\n| Megamedical  | $71.1 \\pm 6.2$   | $65.8 \\pm 6.7$   | $65.7 \\pm 6.8$   | $67.4 \\pm 6.9$   |\n| Synthetic    | $56.3 \\pm 4.9$   | $45.8 \\pm 5.0$   | $44.3 \\pm 5.4$   | $42.7 \\pm 5.5$   |\n\n\n**Why is SAM a consistent second while not trained on medical data?** \n\nThis is an excellent question, for which we don’t have a definitive answer though similar behavior has been observed in the literature[120]. We hypothesize that SAM is better because it was trained on a wider variety of labels and images. Therefore, SAM is less prone to task-overfitting compared to other models. SAM might also be more adapted to the type of interaction used to produce the predictions: clicks. We will make this point in the paper.\n\n**How were M and K selected for the main results?** \n\nWe will clarify our choices of M=8 (number of maps) and K=20 (number of candidate labels) in the main paper. As Appendix Figure 15 shows, performance on the development datasets saturates at M=8 and K=20, with only marginal gains achieved by further raising M. We therefore chose to use these parameters as a balance between performance and the number of structures clinical users might actually expect in practice. Importantly, these parameters were chosen independently of the held-out test datasets in Figure 12.\n\n**How does specifying M and K compare to providing a hierarchy of protocols to existing multi-protocol segmentation models?**\n\nSpecifying M and K is faster than defining a hierarchy of protocols. To demonstrate this, we have built a simple user interface that contains 2 slider bars (one for M and one for K). As the user varies them, the segmentations proposed by Pancakes can adjust in nearly real time (enabled by our lightweight architecture) and propose consistent protocols for all the elements in the set. A protocol can be selected through simple click. Since figures are not permitted in the rebuttal, we cannot share a figure showing this interface, but we will release it with the paper. This alleviates the need to manually annotate many labels and define a hierarchy a priori, especially when there is no clear hierarchy between labels. \n\n**Clarification on comparison to other universal and multi-protocol segmentation methods** \n\nWe compare our model to [45] and [98] in automatic segmentation mode. Although we cite as much relevant work as possible, most are not directly applicable to the new problem setting that we tackle. For example, baselines that are data-specific or that require an explicit hierarchy are not applicable because of their limitations or heavy annotation requirements. As is often the case in medical imaging, we assume that only binary labels are available in our data. Similarly, baselines that  require multiple interactions (clicks, bounding boxes, scribbles) for each image to be segmented  are not applicable to our problem because our goal is to alleviate the burden on the user. We will clarify this choice of baselines in the revised manuscript. \n\n**Changing Bar plot to Table** \n\nThank you for pointing this out, we will include a table for a clearer distinction between the different methods."}, {"review_id": "yoaNsEXoDC", "reviewer": "Reviewer_nZep", "summary": "The paper introduces a novel approach to biomedical image segmentation. Instead of requiring users to define a specific segmentation protocol, Pancakes automatically generates multiple complete and semantically consistent segmentation maps, each representing a plausible protocol across similar images.\nKey contributions include a sampling mechanism to generate diverse segmentations from a learned distribution, a loss function that enforces semantic consistency across images within the same domain, and a lightweight and efficient model that generalizes well to unseen domains, outperforming existing methods such as SAM and MedSAM.", "questions": "- While the model produces diverse outputs, could there be a mechanism (e.g., visual summaries, textual prompts, or clustering) to help users choose the most relevant protocol?\n- Involving clinicians or experts to judge the plausibility or utility of the predicted protocols would greatly strengthen the practical value of the model.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The central idea of ensuring that the same label within a given protocol refers to the same anatomical structure across images in a dataset;\n- Strong generalization to unseen domains and datasets;\n- The novel loss function that rewards the most accurate prediction while encouraging consistency across samples within the same domain.\n- While the concept of generating multiple segmentations per image has been explored in the literature, Pancakes is the first to explicitly enforce semantic consistency across these segmentation protocols. This opens up new directions for developing segmentation models with that on mind.\n\nWeaknesses:\n- Difficulty in assessing how many of the generated protocols are truly meaningful due to the \"best-match\" evaluation strategy\n- From my understanding, although Pancakes produces multiple segmentation protocols, users cannot directly specify or guide the type of segmentation they want. This can limit its applicability in certain workflows.", "was_revised": true, "final_justification": "I liked this paper from the beginning. It is well-made, clearly justified, with interesting results and applicability.", "rebuttal": "We thank the reviewer for their thorough and useful feedback. We are glad they found the task novel and appreciated the strong performance and efficiency of our model, and found the paper clear. We carefully address the remaining questions on the protocols and the choice of baselines.\n\n**Could there be a mechanism (e.g., visual summaries, textual prompts, or clustering) to help users choose the most relevant protocol?** \n\nPancakes produces multiple label maps that are consistent for a set of images to segment. Currently, if the user cares about a specific region, they can choose the most relevant protocol by clicking on that corresponding label map prediction for one of the images. That protocol gets used for all the images in the set.\n\n*User Interface 1*: During the rebuttal phase, we built a simple user interface that contains 2 slider bars (one for M and one for K). As the user varies them, the segmentations proposed by Pancakes can adjust in nearly real time (enabled by our lightweight architecture) and propose consistent protocols for all the elements in the set. \n\n*User Interface 2 -- Going further*: If the user is interested in a specific region, we have designed an alternative UI where the user can specify a region they are particularly interested in by clicking, and only the labels for this specific region will be displayed.\n\nThe rebuttal guidelines prohibit links, but we will make a Jupyter Notebook notebook supporting this available with the final paper to demonstrate the ease of use.  \n\nWhile we have not explored multimodal prompts, we agree that including visual summaries and text prompts is an interesting opportunity for future research. \n\n\n**How can users directly specify or guide the type of segmentation they want** \n\nBeyond the strategies mentioned above, choosing fewer labels per map (lower K) will tend to produce coarser labels. Figure 6, Figure 7 and Figure 10 (in the supplemental material) provide an intuition of how the label maps  predicted change with M and K. During the rebuttal period, we built a simple user interface to demonstrate this effect, containing 2 sliders (for M and K). As the user adjusts them, the segmentations proposed by Pancakes adjusts in near real time (possible due to our lightweight architecture). Once M and K values Pancakes segments consistent segmentations on all images in the set. The desired protocol m or label (m, k) can then be selected through a simple click for a single image. While the NeurIPS change in rebuttal policy to disallow pdfs prohibits us from showing the interface, we will show it in the final manuscript. \n\n**Has a clinical expert evaluated the protocols?** \n\nTo evaluate the accuracy of our predictions, we used expert-provided segmentations for the held-out datasets (unseen at training time). We did not conduct a user study in which clinical experts evaluated the distribution of protocols. Such a study would be useful, but presents design challenges and would require a substantial investment of time by independent experts.\n\n**How many of the protocols are useful?**  \n\nThis is an excellent question. To simulate this, we evaluate Pancakes on a subset of Total Segmentator, a held-out dataset that contains segmentations for many structures, with maximum 8 protocols (i.e., M = 8 ). We find that for each set, all 8 protocols would have some label maps that serve as the best candidate for some ground truth labels, meaning all protocols are used if we consider a holistic collection of possible target labels."}, {"review_id": "nT4pjp3O1u", "reviewer": "Reviewer_jNMX", "summary": "The authors proposed a method that can segment biomedical image with different protocols, named Pancakes. It is framework that can automatically generate multi-label segmentation maps for multiple plausible protocols given a new image from a previously unseen domain. Although the idea is interesting, there are some major concerns about the validity of the measurement, and if this is really clinically meaningful.", "questions": "1. One of the most important concepts of biomedical image segmentation is to accurately segment the targeted tissue/organ/region that have been previously clinically well defined. Only in this way, the segmentation map can be useful for clinical downstream task, like evaluation fat ratio, organ size, cancer aggressiveness, etc. Although the Pancakes can provide multiple different segmentation maps from potential protocols, then who should confirm the clinical meaning of these protocols. And also, how we can utilize the segmentation maps in the downstream tasks? For example, you get a segmentation map, and it looks like for fat segmentation, can you CONFIDENTLY STATE this is indeed for fat segmentation, but not mis-segmented muscle segmentation? If you can, what is your reasoning behind? If you cannot, then I would say this is a relatively big problem.\n2. “The images within each dataset are also split into training, validation, and test splits.” Did you split the dataset based on images, or based on patients?\n3. “UNet-like architecture, with convolutional layers of 32 features followed by PReLU activation”, surprisingly, the original UNet can perform better than other baseline approaches. Like in Table 1, the comparisons are very eye-catching, and a bit not that realistic to me. I would like to ask the authors to dig into why this can happen and provide corresponding clues.\n4. “We start by assessing if any of the predicted protocols produce a label that closely matches a ground truth label.” It is “closely matches” defined by Dice score enough? Is this assessment clinically verified by the corresponding specialists? For example, vessels, they are more caring about connectivity/topology, etc.\n5. “Finally, we only record the label that performs best, identified by a specific map m and label k. We call this Set Dice” This is the problem I mentioned in the first bullet point. How can you use this outcome in downstream clinical task is a really big problem.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The authors noticed an interesting question in biomedical image segmentation domain, and proposed a framework trying to resolve it. \n2. Relatively enough number of medical imaging and organ/tissue were evaluated.\n\nWeaknesses:\n1. The authors seems to ignore the clinical meaning of the segmentation masks, which makes the work a bit not practical. One of the most important concepts of biomedical image segmentation is to accurately segment the targeted tissue/organ/region that have been previously clinically well defined. Only in this way, the segmentation map can be useful for clinical downstream task, like evaluation fat ratio, organ size, cancer aggressiveness, etc. Although the Pancakes can provide multiple different segmentation maps from potential protocols, no one can confirm its clinical usefulness. \n2. The author proposed a new measurement “Set Dice”. It is kind of very adhocly defined, and I have concerns on the validity of this proposed measurement. Related to this, if you check Table 1, you can observe that the UNet authors are using actually performed way better than other well-known baselines under this measurment, which I would assume is another clue about my concern.", "was_revised": true, "final_justification": "The authors resolved my comments well, and thus rating raised.", "rebuttal": "We thank the reviewer for their thorough and useful feedback. We are glad they found the paper interesting, and appreciate the number of tasks we train and evaluate on. Below, we address the reviewer’s questions on the use case, the protocols M and candidates K, the choice of baselines, and the metric used.\n\n**Use case clarification. How can we utilize the segmentation maps in the downstream or clinical tasks?** \n\nWe will clarify that Pancakes is not intended to replace existing clinically validated segmentation protocols. If there is an existing tool for a particular protocol, we would advise using it. Rather, our goal is to support a new capability: enabling biomedical researchers and clinicians to explore a diverse set of plausible, semantically consistent segmentations for an unseen collection of images. After Pancakes has generated the label maps, a researcher or clinician selects which of the proposed protocols best aligns with their intended downstream use (e.g., anatomical volume analysis). We envision two broad classes of use:\n\n(1) *Rapidly segmenting new protocols*. New protocols are frequently introduced [133, 136, 137], and there is a need to produce corresponding segmentations. If a scientist has a particular protocol in mind, but there are no existing tools for segmenting it, they can choose the protocol from Pancakes that best aligns with their intended use. This choice will often suffice. In cases where Pancakes’ segmentations are not sufficiently accurate–for example, when targets are more out-of-distribution than those seen in training–the segmentation maps can offer excellent initializations to interactive segmentation systems such as ScribblePrompt [120]. To demonstrate this, we performed a preliminary experiment and found that:\n\n\n- On average, Pancakes’ predictions can be improved by 5 Dice points with a *single interactive click*, for set sizes larger than one.\n\n- On average, using ScribblePrompt with a Pancakes-initialized segmentation reduces the number of required interactions by half. If used alone, it takes ScribblePrompt 5 to 8 clicks to reach optimal quality predictions, while, when initialized with Pancakes’ predictions, ScribblePrompt can reach them in 3 to 4 clicks.\n\nSince  figures are not permitted in the rebuttal, we cannot provide plots for this experiment, but will include them in the manuscript.\n\n(2) *Exploratory population analysis*. Pancakes will support users in discovering or selecting segmentation strategies appropriate for their scientific or clinical questions. For example, a clinical scientist who studies how anatomy relates to some outcome (e.g., progression of a disease) or predictor (e.g., genetics) can use Pancakes to quickly extract segmentations of multiple candidate anatomical regions that have never been segmented, compute their volumes, and test correlations with clinical outcomes thereby identifying promising candidate regions.\n\n**Is this assessment clinically verified by the corresponding specialists? Who should confirm the clinical meaning of these protocols?** \n\nPancakes is intended for biomedical experts. We assume that as they use the tool, they can identify the mapping between a label and a known structure. To evaluate the accuracy of our predictions, we used expert-provided segmentations for the held-out datasets (unseen at training time). We did not conduct a user study in which clinical experts evaluated the distribution of protocols. Such a study would be useful, but presents design challenges and would require a substantial investment of time by independent experts. \n\n**Did you split the dataset based on images or based on patients?** \n\nWe split the dataset based on patients, and ensured that there is no train/val/test cross-contamination. We will clarify this in the paper.\n\n**Surprisingly, the original UNet can perform better than other baseline approaches.**  \n\nWe are unsure of what the reviewer means by Dice performance Table 1, as Table 1 relates to speed at inference instead of performance on Set Dice score. We will, therefore, address both: \n- *High UNet performance*: There is strong evidence that UNet-based architectures match transformer-based architectures in terms of performance for medical image segmentation [134, 135]. We chose to demonstrate that Pancakes works with even a simple solution, which enables fast throughput and a low number of parameters helps limit overfitting, and especially task overfitting, where the model remembers the structures to segment instead of proposing diverse segmentations. We also highlight that our proposed framework is not dependent on the choice of the model $f_{\\theta_f}(.)$ that infers the distribution parameters $\\phi$. The UNet implementation of $f_{\\theta_f}$ could be replaced with another network, such as the SAM featurizer. We will discuss this in the revised manuscript.\n- *High Set Dice*: Since our method tackles a new task, other models are not designed to perform well on the aspects captured by these metrics (accuracy and consistency). We find that when just looking at accuracy, Pancakes is comparable to SAM and outperforms the other baselines. However, SAM cannot provide semantically consistent segmentations, which leads to substantial performance degradation when considering several images jointly. \n- *Inference Speed*: Unlike SAM-type models in automatic mode, Pancakes remains unburdened from transformer-based architecture and heavy post-processing scripts. It can run inferences for M protocols (of K labels max) in a single forward path, which guarantees much faster inference than our baselines. \n\n**Intuition behind Set Dice** \n\nMost traditional metrics fail to capture what we care about: simultaneous accuracy and consistency. Given a known label, we want to identify if it is present in our protocols *consistently*. For example, if we identify that label *(m=3, k=5)* corresponds to a given structure in one image, we want to ensure that the label *(m=3, k=5)* corresponds to the same structure in other images from the same dataset. The proposed Set Dice captures this effect by selecting the same label map for all images in the set and taking the best performing label averaged across the set. This metric choice closely approximates real-world use-cases where the user selects a preferred label consistently for all images. We also highlight that Best-Candidate Dice metric–which assumes that a user identifies the most accurate label–is frequent in the medical image literature, especially when labels are uncertain [9, 57, 83].\n\n**Additional metrics** \n\n- Note that the value for set size S=1 in Figure 8 corresponds to the traditional Dice score metric. \n\n- Following the reviewer's feedback, we included additional metrics. In addition to Set Dice, we now include Set Intersection over Union (IoU) and set surface distance (percentile 95%), \n\n*Experimental results.*\nWe find that for set surface distance, Pancakes outperforms all the baselines. For set IoU, Pancakes performs similarly to SAM for S=1 (looking at a single image, equivalent to the traditional IoU metric) and outperforms the baselines in all the other cases.\n| Surface Distance          | Set Size = 1     | Set Size = 2     | Set Size = 3     | Set Size = 5     |\n|----------------|------------------|------------------|------------------|------------------|\n| MedSAM         | $16.5 \\pm 2.0$   | $24.2 \\pm 2.5$   | $30.3 \\pm 2.3$   | $36.7 \\pm 1.9$   |\n| ScribblePrompt | $21.5 \\pm 8.5$   | $26.9 \\pm 6.2$   | $31.9 \\pm 6.0$   | $34.4 \\pm 4.1$   |\n| SAM            | $15.9 \\pm 6.5$   | $26.9 \\pm 3.2$   | $32.6 \\pm 2.3$   | $37.6 \\pm 1.9$   |\n| Pancakes       | $8.2 \\pm 1.3$    | $10.6 \\pm 1.8$   | $12.0 \\pm 2.1$   | $13.0 \\pm 2.5$   |\n\n| IoU          | Set Size = 1     | Set Size = 2     | Set Size = 3     | Set Size = 5     |\n|----------------|------------------|------------------|------------------|------------------|\n| MedSAM         | $15.7 \\pm 5.5$   | $11.4 \\pm 4.0$   | $9.9 \\pm 3.7$    | $8.3 \\pm 3.5$    |\n| ScribblePrompt | $43.2 \\pm 7.7$   | $30.1 \\pm 5.1$   | $27.1 \\pm 4.4$   | $23.4 \\pm 3.4$   |\n| SAM            | $52.0 \\pm 7.0$   | $33.9 \\pm 4.0$   | $27.5 \\pm 3.0$   | $22.1 \\pm 2.7$   |\n| Pancakes       | $51.9 \\pm 6.1$   | $47.8 \\pm 6.5$   | $48.8 \\pm 6.2$   | $49.7 \\pm 5.8$   |\n\n\n\n[133]: Merdietio Boedi, Rizky, et al. \"_3D segmentation of dental crown for volumetric age estimation with CBCT imaging._\" International Journal of Legal Medicine 137.1 (2023): 123-130. \n\n[134]: Isensee, F. et al. (2024). _nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation._  In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15009. Springer, Cham. \n\n[135]: Carsen Stringer and Marius Pachitariu. _Transformers do not outperform cellpose._ bioRxiv, pp. 2024–04, 2024 \n\n[136]: _TopBrain 2025 Grand Challenge_ \n\n[137]: Greve, Douglas N., et al. \"_A deep learning toolbox for automatic segmentation of subcortical limbic structures from MRI images._\" Neuroimage 244 (2021): 118610."}]

Author rebuttal:

Meta-review: Authors present a method to generate multiple consistent parcellations
for a given image as well as a given image set. To this end, they
introduce a network and a random segmentation coding similar to
positional encoding. The network produces a representation that can
produce multiple parcellations and then a shallower network converts
the representation to different segmentation maps based on the random
encoding. The details of this process however, are not very clearly
described in my opinion. The sampling process, the architecture of the
final network - how is it able to output variable number of output
segmentation maps is unclear - as well as the computation of the
loss - in particular how Equation 6 and 7 enforces consistent
segmentation across samples - requires further details. 

The paper's goal is to be able to segment any image and generate
multiple segmentation possibilities for the same image. These
segmentations seem to be dense - the method outputs a representation
per pixel and these are mapped to segmentations for each pixel based
on the random protocol.

The main strength of the article is the way the task is defined. The
task itself and how the segmentation is formulated is interesting
indeed. 

The main weaknesses the reviewers highlight are: (1) lack of clarity
in the explanations, (2) unclear motivation and use cases, and (3)
experimental design, especially lack of comparisons with task-specific
segmentation models and introduction of a new metric.

Despite the weaknesses, reviewers seem to have valued the innovation
in the paper.

**Summary**: The article starts with an interesting and novel task
definition. Overall the method seems interesting as well. The clarity
of the descriptions are not very high quality. Authors tried to remedy
this issue in their rebuttals. The use-case of the method is not very
clear. While authors tried to remedy this issue in the rebuttal, I do
not see a clear answer. Despite the weaknesses, the reviewers value
the novelty of the paper and think this would be a good
contribution. I also think it may start some interesting discussions.
Decision: Accept (poster)