Learning General-purpose Biomedical Volume Representations using Randomized Synthesis
Authors: Neel Dey
Benjamin Billot
Hallee E. Wong
Clinton Wang
Mengwei Ren
Ellen Grant
Adrian V Dalca
Polina Golland
Venue: ICLR 2025 Poster (2025)
Keywords: synthetic data
representation learning
medical image analysis
image registration
image segmentation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We develop a synthetic data-driven contrastive learning framework to train a general-purpose 3D biomedical vision model that achieves state-of-the-art results in 3D multimodal image registration and few-shot segmentation.

Abstract: Current volumetric biomedical foundation models struggle to generalize as public 3D datasets are small and do not cover the broad diversity of medical procedures, conditions, anatomical regions, and imaging protocols. We address this by creating a representation learning method that instead anticipates strong domain shifts at training time itself. We first propose a data engine that synthesizes highly variable training samples that would enable generalization to new biomedical contexts. To then train a single 3D network for any voxel-level task, we develop a contrastive learning method that pretrains the network to be stable against nuisance imaging variation simulated by the data engine, a key inductive bias for generalization. This network's features can be used as robust representations of input images for downstream tasks and its weights provide a strong, _dataset-agnostic_ initialization for finetuning on new datasets. As a result, we set new standards across _both_ multimodality registration and few-shot segmentation, a first for any 3D biomedical vision model, all without (pre-)training on any existing dataset of real images.

Published as a conference paper at ICLR 2025
LEARNING GENERAL-PURPON BIOMEDICAL VOLUME REPRESENTATIONS USING RANDOMIZED SYNTHESIS
Neel Dey' Benjamin Billot' Hallee E. Wong'
Clinton J. Wang'
Mengwei Ren"
P. Ellen Grant Adrian V. Dalca'.3
Polina Golland'
MIT CSAIL 2 New York University
Harvard Medical School
deyecsail edu
ABSTRACT
Current volumetric biomedical foundation models struggle to generalize as public 3D datasets are small and do not cover the broad diversity of medical procedures, conditions, anatomical regions, and imaging protocols. We address this by creating representation learning method that instead anticipates strong domain shifts at training time itself. We first propose a data engine that synthesizes highly variable training samples that would enable generalization to new biomedical contexts. To then train a single 3D network for any voxel-level task, we develop a contrastive learning method that pretrains the network to be stable against nuisance imaging variation simulated by the data engine, a key inductive bias for generalization. be used robust representations of input images for
This network's features downstream tasks and its weights provide a strong, dataset-agnostic initialization for finetuning on new datasets. As a result, we set new standards across both multimodality registration and few-shot segmentation, a first for any 3D biomedical vision model, all without (pre-)training on any existing dataset of real images.
INTRODUCTION
Biomedical vision models trained on imaging studies with fixed protocols rarely generalize to new pop- ulations, medical procedures, and imaging devices. These domain shifts then necessitate practically infeasible reannotation and retraining cycles, especially for adaptation to new tasks. Further, volumet- ric annotated biomedical datasets are especially limited sample size and focused on specific medical procedures, diseases, or scales of anatomy, leading current networks to overfit to small subset of biomedical tasks. To overcome this data scarcity and heterogeneity, we present a representation learn- ing framework driven by a synthetic data engine. Our approach yields generalist 3D network that performs well on diverse voxel-level tasks across a range of unseen biomedical contexts in radiology.
Current biomedical foundation models are trained by aggregating publicly available datasets cover multiple domains (Butoi et al., 2023; Chen et al., 2024; Liu et al., 2023a; MH Nguyen et al., 2024; Pachitariu & Stringer, 2022; Xie et al., 2022). However, persistent gaps hinder their widespread adoption. For example, some methods operate only on specific modalities and regions that can be tractably scaled up in sample size, such abdominal CT (Hamamci al., 2024: Li al., 2024a), and thus cannot learn general representations for most other domains, such as in utero fetal MRI. Others treat 2D slices of volumetric images as independent data points (Butoi et al., 2023; Ma & Wang, 2023), often constructing training sets with high inter-sample correlation and yielding models that fail to produce consistent 3D results, Further, existing foundation models almost exclusively focus on seg- mentation and classification and neglect other key vision tasks such as registration. To our knowledge, no biomedical vision foundation model has been demonstrated for multiple disparate 3D tasks yet.
Contributions.
This paper makes advances on two fronts. To gain robustness to large domain shifts in downstream deployment, we first propose a biomedically informed data engine whose samples encom- pass wide range of appearances and semantics. This engine uses randomly sampled spatial config- urations of biomedical shape templates to synthesize images with arbitrary resolutions, appearances, imaging physics, and crucially, minimal influence from any existing biomedical dataset. Unlike train- ing on samples from GANs or diffusion models, which are limited to reproducing only their original training distribution, our engine synthesizes highly diverse samples useful for arbitrarily new biomed-

---

Published as a conference paper at ICLR 2025
volurnes
Domain-agnostic representations produced network (visualized as ouipui channels)
Abnoominal MRI-CT (same sub))
Modality, pose & FOV shifts
CT
Fiitt
Feiai MRI (different silt))
Anatomy posc shifts
Eudy
Cardlac CT-MRI (dilferent Sub)
Modaliy: pose & anatomy shifts
MRI
Figure 1: Representations produced by our framework, trained only on synthetic data, are approximately stable across imaging modalities, field-of-views, and poses on real unseen volumes from various datasets. For each anatomical region (rows), we show two example volumes with substantial variation (col. 1) and six arbitrarily selected network output channels (cols. 2-7) that illustrate this stability. These features and network weights can be used for several voxel-level tasks.
ical contexts that we do not have training data for. We then develop contrastive learning framework that uses paired samples from the data engine to pretrain network for general voxel-level tasks in radiology using an inductive bias of approximate stability to nuisance imaging variation that does not change image semantics, a key property for generalization across datasets (Gruver et al., 2022).
Our experiments demonstrate that the resulting features and weights enable broad generalization on the key biomedical tasks of 3D registration and segmentation across several diverse datasets. We achieve state-of-the-ar unsupervised multimodality image registration by simply using the network's approximately appearance invariant and pose equivariant representations (Fig. 1) drive existing registration solvers. The proposed network can also be used as an off-the-shelf dataset-on -anasst-agnostic initialization for finetuning any voxel-level task. Specifically, we demonstrate strong few-shot segmentation performance across several highly variable downstream datasets, thereby removing the need for the cunbersome dataset-specific self-supervised pretraining commonly used today. Code, tutorials, and model weights are available at https:/ //www neeldey com/ anat omix/.
2 RELATED WORK
Generative image models. Learning- Learning-based generative models (Brock et al., 2018; Goodfellow et al., 2014; Karras al., 2020; Luo, 2022; Rombach et al., 2022; Song et al., 2020) trained on internet-scale natural vision sets (Schuhmann et al., 2022) can now synthesize photorealistic samples for pretraining general networks (Donahue & Simonyan, 2019; Fan et al., 2023; Li et al., 2023; Tian et al., 2024b). However, such generative models trained instead on the few thousand publicly available anatomy- and modality-specifii annotated volumes in biomedical datasets (Baid et al., 2021; LaMontagne al., 2019; Qu al., 2024; Wasserthal al., 2023) generally do not learn representations that can
2

---

Published as a conference paper at ICLR 2025
(A) Label Ensemble Model
Segmentattons from the TolelSegmertlttrr dalaset
Rondomly sampled 3D shape templates
3D shope ensemble
Vertebrae
Compose
Liver
Autochchon
Lake
Rias
Calen
(B) Appearancs Model
shape
Randomly sampled intensities
Randornly sampled sugmentations
ArTennso Keston
GMM
Garera
Perlin
(C) Central slices of volumntric training samoles (voxei-level annotations not shown)
Figure 2: Data engine. A. We randomly sample binary labels as templates from a large database of anatomical segmentations to create 3D label ensembles of randomly deformed templates. B. Given synthesized label ensemble and an appearance model, we synthesize two volumes pretrain network with a dense multi-view contrastive objective. C. Example synthetic training volumes produced by our data engine. These samples are not intended be necessarily realistic, but rather to serve as diverse and useful training data for learning general tasks in arbitrary radiological domains.
generalize new biomedical domains. contrast, the label synthesis component of our data engine draws loose inspiration from the Dead Leaves model (Baradad Jurjo et al., 2021; Lee et al., 2001). This hand-crafted generative model considers images to be compositions of randomly deformed shape templates (such as cubes, ellipsoids, etc.) with static intensities capture the statistics of natural images. We compose biomedical shape templates similarly but develop several further extensions and propose a distinct appearance model, as explained below.
Domain randomization. To learn robustness to domain shifts at deployment, domain randomized generative models (Tobin al., 2017) trade realism for diversity when generating training data for downstream models. For example, domain randomized methods for brain segmentation (Billot et al., 2023a;b; Gopinath et al., 2023; Hoopes al., 2022b) and registration (Hoffmann et al., 2021: Iglesias, 2023) train on synthetic brains simulated from label maps and require large collections of expert brain annotations. Recent work dataset-agnostic instance segmentation (Chollet al., 2024: Dey et al., 2024) generalizes beyond brains by simulating both annotations and images using pre-specified shape prior. We build on these concepts to develop task-agnostic data engine that simulates images with highly variable appearances and physics using compositions of biomedical shape templates.
Invariant imaging features. Given the heterogeneous nature of biomedical imaging protocols, several existing strategies aim to extract features robust to nuisance variation. When registering images across modalities, aligning modality-invariant hand-crafted local descriptors (Heinrich et al., 2012; 2013) and/or edges (Haber & Modersitzki, 2006) is common. With deep basedd multimodality registration, this inter-modality invariance can be learned (Dey et al., 2022; Mok al., 2024; Pielawski et al., 2020), leading improved performance at the cost of dataset-specific training. Beyond registration, spanifie invariant features have been learned by exploiting large repositories of annotated brains (Chua & Dalca, 2023; Liu et al., 2023b; 2024). Our work obviates

---

Published as a conference paper at ICLR 2025
Vol.
Dec00et L
Volume Synthesis
U-Net Encoder
Decoder
Label
Anchar Posilive
super-super Multi-scale Contrastive Loss
Vol. 2
Decoder 1
Le
Volume Synthesis
U-Net Encoder
Figure 3: Representation learning, Given 3D label map and two corresponding synthetic volumes sampled from our data engine, we process them using single UNet with shared weights. The UNet is pretrained contrastively at each decoder layer: for randomly sampled anchor, features sampled from the same label in both volumes serve as positives, while features from other labels act as negatives.
the need for dataset-specific training, anatomical region-specific modeling, and large-scale annotation collection by extracting modality- and appearance-invarian features in an amortized manner.
Volumetric pretraining. Many methods pretrain unannotated images from a dataset before supervised finetuning on small labeled subset. These methods often employ self-superrvsed reconstructive (Chen al., 2019a; Tang et al., 2022: Valanarasu al., 2024; Zhou al., 2023; 2021) and/or discriminative (Chaitanya et al., 2020; 2023; Ren et al., 2022; You et al., 2024) losses. However, these pretraining strategies exploit heuristics about their target datasets that often do not broadly generalize (Dong et al., 2021; Ren et al., 2022). Our approach instead yields network that generalizes to arbitrary radiological datasets and does not require bespoke pretraining frameworks for each project. Lastly, recent biomedical foundation models trained pooled datasets of 2D slices (Butoi et al., 2023; Ma & Wang, 2023; MH Nguyen et al., 2024; Wong et al., 2023) generally require interaction (via bounding boxes, scribbles, etc.), struggle with 3D consistency, and restricted to segmentation. We instead directly train for general tasks using synthetic 3D volumes and do not work on interactive tasks.
METHODS
This section first details the proposed data engine (Fig. 2), then describes the representation learning strategy (Fig. 3), and concludes with applications towards 3D registration and segmentation.
Data engine: label ensemble model (Fig. 2A). We create synthetic 3D label ensemble volumes by sampling from repository of biomedical shape templates. As templates, ~45,000 binary volumes from the TotalSegmentator dataset of 104 annotated organs in 1 1,204 CT vol- umes (Wasserthal al., 2023). For each label ensemble volume, iteratively populate 3D volume with random number of randomly sampled templates that are each then deformed and assigned label corresponding the sampling iteration. As radiological volumes are often surrounded by empty space (e.g., air), we simulate foreground and background separation by applying a binary foreground mask to two-thirds of the synthesized label ensembles by multiplying them with a randomly deformed binary sphere with random radius and center. Further, as many radiological applications and domains have layer-like structures the randomly encase half of the foreground-maske volumes within envelope labels of random widths.
the freely available
interface (e.g., fat and skin), we
appearance model (Fig. 2B). Given a 3D label ensemble with K labels, we sample the intensities of two volumes Vi and V2 from two independent K K-component Gaussian mixture models (GMMs) each with parameters {MKI, of k and {uz.. o 1 1 respectively, all of which are randomly drawn from uniform distributions. For each spatial index in L with label k sample the initial intensities in Vi and 12 from A N(H1) of and N (μz2 2 respectively. We then pointwise multiply them with Perlin-like noise (Perlin, 1985) to simulate spatial texture and augment
Data engine:
4

---

Published as a conference paper at ICLR 2025
Moving Source
ANTs-M MI
SynthMorph
ConvexAdam
Ours
Fixed Target
Abdominal MRUCT
+ Label Overlay
Label Overlay Cardiac MRICTT
Figure 4: Multi-modality 3D registration. Using our representations with the Convex registration solver ("Ours") yields accurate alignment of challenging pairs intra-subject abdominal MRI-CT (top) and inter-subject cardiac MRI-CT (bottom) with large deformations across modalities.
using transformations relevant to biomedical volumes such as random bias fields, Fourier-spikes, Gamma shifts, blurring, Gibbs ringing, resolution degradations, noise, motion, flips, and affine warps. All intensity augmentations are sampled independently but the geometric augmentations are shared.
In summary, we synthesize 3D label ensemble and draw volumes and 1/2 from it that differ in appearance but share 3D semantic layouts. This is repeated with randomized hyperparameters for each sample generate a synthetic dataset. Low-level modeling details provided App. B.
Contrastive pretraining (Fig. 3). As our data engine provides exact label supervision, we develop a representation learning loss that is spatial extension of multi-positive supervised contrastive learn- ing (Khosla et al., 2020). To pretrain network F IR NW x + IR H XW xDxC where H, W, and D are spatial dimensions and C is the number of output features, we use inductive bias voxel-level features within 3D shape having similar spatial representations, regardless of differences in appearance. We assume that an anchor spatial index / (where = {l...., 2HW D}. i.c., voxels pooled from Vi and V2) with features fü IR in label & should have similar representations to all other indices label labels. As in Chen al. (2020a), use non- monelinnar projection Z R "I DXC - R Nx DxCz on F's outputs followed by an L2 normalization when computing the contrastive loss,
both F F(V) and F(V2) and dissimilar representations to indices from other
exp(z: 2p/T)
C
log
EgEQ()) exp(2, Zg/T)'
NEI
PEP())
where: RR22, 7 hyperparameter, Q(1) = [if (i.e., non anchor spatial indices), and P(i) is the set of all positives for anchor i, s.t. P(i) {p E Q(i) kp = h;}. where ki is the label of spatial index æ. Lastly, we use this loss on multiple decoder layers of F during training leverage multiscale superriipoi in Dey et al. (2022); Park et al. (2020); Ren et al. (2022).
Pretraining implementation details. We implement as four-level 3D convolutional UNet (Ron- neberger al., 2015) following the architecture from (Ren al., 2022) and construct Z as a 3-layer So--nodewwide MLP. While F' can be any volume-to- -o-ouumm network, we use U-Net as the standard architecture for biomedical imaging tasks and performs well across datasets (Isensee et al., 2024; Stringer & Pachitariu, 2024). and are pretrained jointly for 600,000 iterations with a batch size of one 1 1283 label map, each generating two 1283 volumes. We compute the contrastive
5

---

Published as a conference paper at ICLR 2025
(a) Multi-modality registration Dice coefficients
(b) Subject-wise Dice coeff.
Mediara: .38 .12 .43 .60
.51
.78
.73
.84
.51
.62
.58
64
.59
.64
.66
.73
Aottteg-bbtteennno MRCT
Aw.AN Qu--
AWTb-W
(Dice)
Loan2Reg
0.8
ConvexAdam Ours Conveschlam
0.4
SynthMorph shaces
Unnegistered
0.3
unloradiCON
MM Ni-----5
ANTMMI AN Ours
Commesddamm
ConvexAdarm
0.5
Folds 96:
.0%
.0%
.0%
11%
3% 296 .1%
.0% .0%: .3%
11%
.25
.4% .356
0.0
LeannReeg-Abdomen MRCT
MM-WHS
Subject
Figure 5: Multi-modality 3D registration results. (a) Dice boxplots for each method for L2RAb (left group) and MM-WHS (right group), with corresponding medians reported on top of each box and the mean percentages voxels with folds produced by each method reported the bottom; (b) Using our features leads to consistent registration improvements at the subject-level.
loss (with temperature r 0.33) on 512 randomly sampled indices at each iteration for each decoder layer due to memory limitations. Lastly, only used during pretraining and is discarded for all downstream tasks. All other pretraining details are described in App. B.3.
Downstream tasks: multi-modality registration. Gradient- deformable registration objectives typically take the form of Creg d(Vixxed, Vimoving vp) + AReg(sp), where image Vimowing is to be aligned to Vfixed by deformation yp, subject to regularization Reg(-), ) and d(-) (-) is an image dissimilarity score. To align images across imaging modalities, we simply replace the input volumes with our pretrained network's features as in Creg = ((F((axee)) F(Vmoving )op) (oop)+XAu +XReg(pp) This approach is compatible with any existing gegh-performance registration solver, such the ConvexAdam (Siebert et al., 2021; ; 2024) and ANT's (Tustison et al., 2020) frameworks used in our experiments below.
Downstream tasks: few-shot segmentation. For N-label few-shot semantic segmentation, finetune the pretrained network F on a small number of annotated volumes with an additional convolutional layer with softmax activation and N output channels. We optimize the network using an equally weighted sum of the soft Dice and cross-entropy losses (Isensee et al., 2021; Taghanaki al., 2019). To extract strong performance for all baselines in the challenging setting of finetuning on only one or few annotated volumes, we use extensively-tume augmentation pipelines and finetune all layers of each network for a high number of iterations (37,500) with cosine learning rate decay.
EXPERIMENTS
The pretrained network's output features for inter and intra-subject volume pairs with semantically similar content subject to strong domain shifts are visualized in Fig. 1. Below, we present experiments that investigate the utility of our learned representations for multi-modality registration, the network weights as a pretrained initialization for few-shot segmentation, and analyze our modeling decisions.
4.1 UNSUPERVISED MULTI-MODALIT DEFORMABLE 3D REGISTRATION
Data and setup. We use the Learn2eeg AbdomenMRCT (Hering et al., 2021) (L22RAb) and MM-WHS (Gao et al., 2023; Zhuang, 2018; Zhuang et al., 2019) datasets to benchmark MRI to CT volume registration. L2RAb is an abdominal registration benchmarking dataset, whose publicly available portion provides eight affine-aligned intra-subject MRI and CT pairs of size 192 x 160 x 192 at 2 x x 2mm3 resolution, with labels for four organs. MM-WHS originally heart segmentation dataset, contains 20 annotated MRIs and CTs (from distinct subjects) and affine-align all volumes to a common space of size 160 x 160 x 128 at 1.142 x 1.142 x 1. 1.283mm² resolution (see App. B. 4.4 for further details). The registration experiments unsupervised and

---

Published as a conference paper at ICLR 2025
Image
Random
Models Genesis
Disrupt. Autoenc.
PrimGesSeg
Manual Label
Abdominel MRI
Fetal BOLD MRI
Cardisc MRI
Figure 6: Few-shot 3D segmentation qualitative results. All methods (columns 2-7) were fine- tuned on 3, 3, and 1 multi-label annotated volume(s) for each respective dataset (rows 1-3).
we split L2RAb and MM-WHS. into 1/7 and 5/15 validation/testing pairs, respectively, where the validation pair(s) are only used for tuning registration hyperparameters.
Baselines and evaluation. We compare unsupervised and training-free multimodality registration frameworks. Our iterative baselines include the widely-used ANTs library (Avants et al., 2008; Tusti- son et al., 2020) with mutual information loss (Mattes et al., 2001; Wells III et al., 1996) (ANTs-M) MMI) and the state-of-the-ar (Hering al., 2021) multimodality method, ConvexAdam (Siebert al., 2021). We further use two datasct-agnostic registration networks, SynthMorph-shhe shapes (Hoff- mann et al., 2021) and uniGradIcoN (Tian et al., 2024a), with the latter also using optional instance optimization (ur.GraddCoN++T IO)) For evaluation. we report post-registration volume overlap (Dice) anatomical structures. We also assess deformation inverse consistency using the percentage of folding voxels, where dep( < O, for Jacobian J&p of the estimated warp y.. Folding percentages below 0.5% of voxels are generally considered negligible and methods producing higher Dice values while staying under this threshold preferred (Qiu et al., 2021).
Adaptation to use network features. We modify ANTs and ConvexAdam to use our pretrained network's 16 extracted features. We use ANTs' multichannel mode with the MSE loss and tune its hyperparameters heuristically on validation pairs. For ConvexAdam, we concatenate our features with its default handerafted features and perform grid search for both the original implementation and our variant over four hyperparameters. Lastly, the deep learning baselines assume single-channel input volumes and thus cannot directly
multichannel network features.
Results. Figs. 4 and present results on held-out testing pairs. the typically-used ANTS-MA with 26 and 5 points of median Dice improvement L2RAb and MM- WHS, respectively, while maintaining nearly identical low folding characteristics. Further, driven by our network features, ConvexÅdam-Ours. outperforms all methods terms of volume overlap and improves on its base ConvexAdan method by 11 and 6 Dice points, with the same folding ratios. In contrast, the
strongly improves upon
and inra CON+ IO methods perform well on MM-WIH (where all hearts are roughly centered) but cannot handle the larger deformations in 1.2R They do, however, yield nearly diffeomorphie transformations, producing almost zero folds. Lastly, without iterative optimization, uniGradICON demonstrates limited generalization across large intensity- based domain gaps. We note that all methods produce folding percentages that are under the threshold of 0.5% folding voxels. Additional grid search results on the validation sets are in App. A.2.
4.2 FEW-SHIOT 3D MULTI-LABEL SEMANTIC SEGMENTATION
Data and setup. We evaluate fho- segmentation performance on a diverse collection of datasets: cardiac bSSEP MRI from MSD-Heart (Antonelli al., 2022), abdominal CT from AMOS (Ji et al., 2022), prostate T2w MRI from PROMISE12 (Litjens et al., 2014), abdominal SPIR MRI (Akin et al., 2016; Clark al., 2013; Kavur et al., 2019; Linehan al., 2016) from Leann2eeg-Abdmenn (Hering
7

---

Published as a conference paper at ICLR 2025
Table Few-shot 3D segmentation Dice means and their bootstrapped std. deviations. Bolding and underlining represent the best and second bessnd-best Dice, respectively.
Params. MSD-Heart
PROMISE12 L2RAb-MRI
FeTA
AMOS-CT WUFetal
Fine-tuning vols.
2
3
3
Number of classes
1
7
Rand. Init. UNet
5.9M
88(.01)
80(.02)
88(.06)
.78(.3)
.56(.01)
.73(.02)
Transfer Learning
5.9M
88(.02)
82(.01)
82(.06)
.78(03)
.2((01)
.74(.02)
Models Genesis
19.1M
84(.0))
.3(,03)
81(.0)
.79.03)
.55(.11)
.66(.0))
MedicalNet
17.3M
86(.02)
.3(,04)
.73(.07)
.74.04)
.44(.22
.50.02)
PrimGeoSeg
67.2M
88(.01)
.79(.02)
84(.05)
.79(03)
.63(.11)
.76(.22)
SMIT
67.2M
.8(.02)
.72(.03)
88(.0)
.76(04)
.58(.01)
.73(.22)
Disruptive AE
67.2M
82(.02)
.64(03)
.77(.07)
.74(04)
.50(.22)
.70(.2)
Ours
5.9M
89(.01)
.85(00)
.86(.6)
.80(03)
.61(.01)
.76(.02)
Full supervision
5.9M
91(.01)
900,00)
.89(.55)
.3((0))
.85(.00)
.88(01)
Table Multitask capabiliiiis of current 3D biomedical foundation models. Using foundation models as feature extractors for multimodality registration with the ANTs solver, only Ours outper- forms the solver defaults (Mutullnnoo) indicating that other methods are limited to segmentation.
Dataset
Mutuallnfo PrimGeoSeg ModelsGen.
SMIT
DAE
Ours
L22 AbdO
.8(.10)
46(.09
.38(07)
48(.08
550.07) .70(.99)
MM-WHS
.58(.03)
.1(.02)
.1((.3)
53(.03) .53(003) .63(.02)
et al., 2021), fetal brain HASTE MRI from FeTA (Payette et al., 2024), and in-house dataset of whole uterus fetal BOLD MRI (WUFetal), WUFetal provides labels for the placenta, amniotic fluid, and the fetal brain and body. It is included an out-of-distribution test dataset for baselines below that have not been pretrained on fetal images. Lastly, we operate in the few-shot regime, where only 1-3 annotated volumes per dataset are used for finetuning. The dataset splits are in App. B.5.
Baselines. We use 3D foundation models specifically pretrained for multi-label segmentation on mul- tiple datasets. These include the masked autoencodingg-ased Model ls Genesis (Zhou al., 2021), SMIT (Jiang et al., 2022), and Di rrupt ive AE (Valanarasu et al., 2024). Other transfer-learning baselines include MedicalNe: (Chen et al., 2019b) and PrinGeoSeg, the latter being pretrained to segment synthetic binary volumes with simplistic shapes. To explicitly test transfer learning with a matched architecture (Trans ferLearn ingg), we further train a fully supervised UNet with the same architecture as Ours LaMontagne et al., 2019). We also test a randomly initialized UNet (with matched architecture to Ours) trained on few (RandIni UNett) ) or all (Fu.l (FullSSpel SSupervisi wiviion on) volumes in the training split. Current 2D interactive binary segmentation foundation models (Butoi et al., 2023; Ma Wang, 2023) excluded from the experiments as they require data. All methods were finetuned with extensive data augmentation, as described App. B.5.
large- scale neuroimage segmentation dataset (Hoopes et al., 2022a;
prompts and do not apply to 3D multi-label
Results. Table and Fig. present few-shot segmentation results. Our framework produces pretrained weights that consistently improve upon random initialization. Crucially, this improvement achieved using only dataset-agnostic pretrained weights and without any pretraining unlabeled real volumes or data from similar domains, as commonly done in previous work. Compared to larger foun- dation models specifically trained for segmentation, our pretrained general-purpose network achieves the first or second rank consistently and has the best average rank. Importantly, the second-best model changes dataset-tor -o-ttase,, indicating that the baseline methods do generalize consistently to new biomedical contexts. We lastly emphasize that we achieve these gains with fewer parameters, without access to any real images, and while being applicable to other tasks as well, as described below.
4.3 ABLATIONS AND OTHER ANALYSES
Multitask capabilities. Current 3D biomedical vision foundation models are evaluated primarily for semantic segmentation, raising the question of how well their features generalize to other tasks. To
8

---

Published as a conference paper at ICLR 2025
3D input volumes (differenn subjects
1.077
QUTS 1- D20
Qur
0.233
0.89
1-:333
smahapos
1- 0.33
Config, used in our experiment
Varying pretraining labels
Figure 7: Features for varying pretraining configurations. When our framework is trained with different r values (cols. 2-3) or on synthetic data generated from other label sources (cols. 5-6), the network features for real biomedical volumes (col. 1) are degenerate and/or sensitive to nuisance imaging variation. We visualize an arbitrary channel for each model, with more in App. Fig. 11.
test this, we utilize the few-shot segmentation baselines that provide both pretrained encoders and decoders as general-purpose feature extractors and use their features drive generic registration solver (ANTs), with implementation details provided in App. B.4.3. As reported in Table 2, all current 3D biomedical foundation models are unable to extract features that improve upon the baseline setting used by the default ANTs solver (Mutual In) In contrast, our approach (Ours) outperforms by wide margins and is thus the only method to yield multitask general-purye purposs features for the highly disparate tasks multi-modality registration (Fig. 5) and few-shot segmentation (Table 1).
Label generation. To evaluate our label ensemble model, we replace it with other training labels while keeping the appearance model fixed. We test pretraining using: (a) synthetically generated labels that have no biomedical priors (Hoffmann et al., 2021) (smshapes), (b) 1,573 label maps of real brain MRI annotated using the FreeSurfer protocol (Fischl, 2012) (Brains) that represent real anatomical structures with dense ver-voxel annotations, and (c) combination of Brains and our model mix real and synthetic sources of label maps. Details regarding these models are provided in App. B.6.
Table 3: Effect of pretraining configurations on downstream tasks via Dice means and their bootstrapped std. deviations. Row 1 corresponds the configuration used in our previous experiments. Registration experiments are all performed using the ConvexAdam (Siebert et al., 2021) solver.
Pretraining config.
Registration (L.2RAb)
Few-shot segmentation Dice (?)
Pretraining loss T
Labels
Dice(t)
Folds%(L)
WUFetal MSD-Hear
AMOS CT
Ours
0.33
Ours
.74(10)
0.22%
76(.02)
.8(.01)
61((01)
Ablating pretraining labels
Ours
0.33
smshapes
.68(10))
0.20%
.73(,02)
88(00))
.60,01)
Ours
0.33
Brains
57((10)
0.16%
.74(.02
88(.0))
.60(01))
Ours
0.33
Ours-Brains
.71(.88)
0.29%
.76(02)
.9(001)
.61(.01)
Temperature variation
Ours
0.07
Ours
.58(10) .64(09)
0.17% 0.19%
.72(02)
.0001)
.60.01)
Ours
0.20
Ours
.78(.2))
.9(.01)
.62(.01)
Ablating prefraining loss
Denoising
Ours
51(.00)
0.19% 0.24%
58(.02)
83(.03)
.66.01)
Remove labels
Ours
.50(1)
.66(02)
.6(02)
.5((01)
Ablating pretraining augmentations
Ours (Row 1) w/o FG mask Ours w/o PG mask, w/o offline augm.
.63(.9))
0.27%
.73(.02)
86(.03)
.60.01)
.63(.09)
0.22%
.74(.2)
.8((01)
.66(01)
Ours w/o FG mask, w/o augm.
.59(.9)
0.63%
.70(.2)
84(.02)
.1((00)

---

Published as a conference paper at ICLR 2025
Table 3 rows 1 show that both registration and segmentation performance decline with other choices of pretraining labels. Further, combining our labels with Brains does not affect segmentation but worsens abdominal registration. Lastly, pretraining on these alternative label models leads to unstable network features on real data (Fig. 7, right), likely explaining the performance degradation.
Temperature (r). The temperature hyperparameter in the contrastive loss (eq. 1) controls the penalty weight on negative pairs (Wang & Liu, 2021). In natural vision, smaller T values (such as 7 = 0.07 (Chen al., 2020b)) are used upweigh difficult negative pairs in the loss. However, as we train our network to learn similar representations within each label across highly disparate appearances, the negative pairs are all of higher difficulty and require relaxed 7 of 0.33 for stable training. We find that using lower T leads to degenerate aliased features (Fig. 7) and worse registration results (Table 3). Interestingly, segmentation benefits slightly from an intermediate setting of 0.20, indicating a tradeoff between optimal representations for downstream registration versus segmentation.
Pretraining objectives. We now retain our data engine, but pretrain using other frameworks. We compare against matched architectures that are (a) pretrained to denoise the augmentations used our data engine (Denoi s in) as in Iglesias et al. (2023) and (b) pretrained using suff-suuervision on intensities alone without using label information [RemoveLabell
as in Chua & Dalca (2023); Ren et al. (2022). In Table 3, rows 1, 7, and 8, we find that our label- subel-supervised multi-positive contrastive strategy yields the highest results for both registration and segmentation.
Data engine augmentations. We now ablate the augmentations used in the proposed data engine used during pretraining by cumu- latively removing the foreground masking, the augmentations used during offline image synthesis (App. Fig. 12), and all augmentations, such that the training images are simply the Perlin-corrupted Gaus- sian mixture model outputs. In Table 3 rows 9-11, we observe a substantial drop in both registration and segmentation performance without the proposed augmentations.
1.0
AMDS-CT seg. Dice bosploes vo medians
A22
014
.022
0.9
0.8
0.7
0.6
0.5
Random Intiolizztion
Finetuning with more annotations. Finally, while our segmen- tation experiments focus on the few-shot setting, our learned ini- tialization also benefits scenarios with more supervision. Fig. 8 quantifies how finetuning the proposed network with more annotated volumes leads to improved segmentation on AMOS-CT (Ji et al., 2022) relative to random initialization across settings, albeit with smaller improvements given more annotations.
Ours
0.4
45 Iineurinng subjects
180
Figure 8: Fine-tuning perfor- mance as a function of annota- tion budget.
5 DISCUSSION
Limitations and future work. Our approach does have limitations. We pretrained our network to be stable against intensity variations (among other variables) and demonstrated its utility for registration and segmentation. However, a small set biomedical tasks rely on relative intensities (Nakamura al., 2017; Thomalla al., 201 1) and texture (Yu al., 2017), making intensity invariance subop- timal inductive bias for them. Further, while we operate on general volumetric tasks, certain inverse problems like MRI reconstruction take sensor-domain non-Cartesian (Schlemper al., 2019) mea- surements as multichannel inputs, requiring domain architectural changes (Singh et al., 2022). Lastly, our segmentation experiments finetune our pretrained network on specific datasets, potentially introducing complexity for some clinical users. However, future extensions could directly use our proposed data engine to train promptable 3D segmentation models that require no such finetuning.
Conclusions. When combined with the right inductive biases, synthetic data models informed by biomedical templates enable the training of powerful general-purpose networks. This is important for 3D radiology, where existing annotated datasets are limited in sample size, often acquiring only dozens to at most a few thousand volumes. This leads to inflexible models that deteriorate under domain shifts. Trained only on synthetic volumes with our proposed framework, the resulting network provides substantial benefits on a variety of radiological tasks. For example, its representations yield substantial improvements the ofte-of-thhearr in training- free multimodality deformable registration, key area in biomedical vision. Further, the network can also serve as a downstream dataset- initialization for few-shot segmentation tasks and lead to improvements across multiple datasets.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We gratefully acknowledge support from NIH NICHD RO1HD100009, NIH NICHD IROIHD 14338, NIH NIBIB IRO1EB032708, NIH NIBIB RO1EB033773, MIT-IBM Watson AI Lab, MIT Jameel Clinic, and the MIT CSAIL-Wistron Program.
REFERENCES
O Akin, P Elnajjar, M Heller, R Jarosz, B Erickson, S Kirk, and J Filippini. Radiology data from the cancer genome atlas kidney renal clear cell carcinoma TCGA-KIRC collection. Cancer Imaging Archive, 2016.
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature communicatiinss 13(1):4128, 2022.
Brian Avants, Charles L Epstein, Murray Grossman, and James Gee. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurode- generative brain. Medical image analysis, 12(1):26 41, 2008.
Brian B Avants, Paul Yushkevich, John Pluta, David Minkoff, Mare Korczykowski, John Detre, and James C Gee. The optimal template effect in hippocampus studies of diseased populations. Neuroimage, 49(3):2457-2444 2010.
Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Cllpathy-Cramee Felipe C Kitamura, Sarthak Pati, et al. The RSNA-ASNR- MICCAI BRATS 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint 1:Xiv:100.023:: 2021.
Manel Baradad Jurjo, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. Advances Neural Information Processing Systems, 34:2556-2569, 2021.
Benjamin Billot, You Colin, Magdamo Cheng, Sudeshna Das, and Juan Eugenio Iglesias. Robust machine learning segmentation for large-scale analysis of heterogeneous clinical brain MRI datasets. Proceedings of the National Academy of Sciences (PNAS), 120(9):1-10, 2023a.
Benjamin Billot, Douglas N. Greve, Oula Puonti, Axel Thielscher, Koen Van Leemput, Bruce Fischl, Adrian V. Dalca, and Juan Eugenio Iglesias. Synthseg: Segmentation of brain mri scans of any contrast and resolution without retraining. Medical Image Analysis, 102789,,, 2023b.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1009, 2018.
Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Universeg: Universal medical image segmentation. arXiv preprint arXiv:2304.06133 2023.
M Jorge Cardoso, Wenqi Li, Richard Brown, Nic Ma, Eric Kerfoot, Yiheng Wang, Benjamin Murrey, Andriy Myronenko, Can Zhao, Dong Yang, et al. Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.027011 2022.
Kerem Cekmeceli, Meva Himmetoglu, Guney Tombak, Anna Susmelj. Ertunc Erdil, and Ender Konukoglu. Do vision foundation models enhance domain generalization in medical image segmentation? arXiv preprint arXiv:2409.07966, 2024.
Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in neural information processing systems, 33:1 183:12566-1255 2020.
Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation. Medical image analysis, 87:10279 2023.
11

---

Published as a conference paper at ICLR 2025
Liang Chen, Paul Bentley, Kensaku Mori, Kazumari Misawa, Michitaka Fujiwara, and Daniel Rueckert. Self-supervised learning for medical image analysis using image context restoration. Medical image analysis, 18:109,,, 2019a.
Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williammon, Guillaume Jaume. Andrew H Song. Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards general-purpose foundation model for computational pathology. Nature Medicine, 30(3):850- 2024.
Sihong Chen, Kai Ma, and Yefeng Zheng. Mcd3d: Transfer learning for 3d medical image analysis. arXiv preprint arXiv: 1.X1v:19440002... 2019b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10022 2020b.
Etienne Chollet, Yaël Balbastre, Caroline Magnain, Bruce Fischl, and Hui Wang. A label-free and data-free training strategy for vasculature segmentation in serial sectioning oct data. In Medical Imaging with Deep Learning (short paper track), 2024.
Yue Zhi Russ Chua and Adrian V Dalca. Contrast invariant feature representations for segmentation and registration of medical images. In Medical Imaging with Deep Learning, short paper track, 2023.
Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, The cancer imaging archive (TCIA): Maintaining and operating public information repository. Journal of digital imaging, 26(6): 1045 1057, 2013.
Neel Dey, Jo Schlemper, Seyed Sadegh Mohseni Salehi, Bo Zhou, Guido Gerig, and Michal Sofka. Contrareg: Contrastive learning of multi-modality unsupervised deformable image registration. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp.
66-77. Springer, 2022.
Neel Dey, Mazdak Abulnaga, Benjamin Billot, Esra Abaci Turk, Ellen Grant, Adrian V Dalca, and Polina Golland. Anystar: Domain randomized universal star-convex 3d instance segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.
7593-7603, 2024.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in neural information processing systems, 32, 2019.
Mengjin Dong, Long Xie, Sandhitsu R Das, Jiancong Wang, Laura EM Wisse. Robin DeFlores, David A Wolk, Paul Yushkevich, Alzheimer's Disease Neuroimaging Initiative, et al. Deep- atrophy: Teaching a neural network to detect progressive changes in longitudinal mri of the hippocampal region in alzheimer's 's disease. Neuroimage, 243:118514, 2021.
Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. arXiv preprint arXiv:2312.04567, 2023.
Bruce Fischl. Freesurfer. Neuroimage, 62(2):744-781, 2012.
Shangqi Gao, Hangqi Zhou, Yibo Gao, and Xiahai Zhuang. Bayeseg: Bayesian modeling for medical image segmentation with interpretable generalizability. Medical Image Analysis, 89:102889, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014.
12

---

Published as a conference paper at ICLR 2025
Karthik Gopinath, Douglas N Greve, Sudeshna Das, Steve Amold, Colin Magdamo, and Juan Eugenio Iglesias. Cortical analysis of heterogeneous clinical brain mri scans for large-scale neuroimag- ing studies. In International Conference Medical Image Computing and Computer-Assister Intervention, pp. 35-45. Springer, 2023.
Nate Gruver, Marc Finzi, Micah Goldblum, and Andrew Gordon Wilson. The lie derivative for measuring learned equivariance. arXiv preprint 0rXiv:2210.029844 2022.
Eldad Haber and Jan Modersitzki. Intensity gradient based registration and fusion of multi-modal images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 726-733. Springer, 2006.
Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Omer Faruk Durugol, Bastian Wittmann, Tamaz Amiranashvili, Enis Simsar, Mehmet Simsar, Emine Bensu Erdemir, Abdullah Alanbay, Anjany Sekuboyina, Berkan Lafci, Christian Bluethgen, Mehmet Kemal Ozdemir, and Bjoern Menze. Developing generalist foundation models from multimodal dataset for 3d computed tomography, 2024. URL https::////. arxiv. org/ abs/ / 2403. 17344.
Mattias P Heinrich, Mark Jenkinson, Manav Bhushan, Tahreema Matin, Fergus V Gleeson, Michael Brady, and Julia A Schnabel. Mind: Modality independent neighbourhood descriptor for multi- modal deformable registration. Medical image analysis, 16(7):1423-143: 2012.
Mattias Paul Heinrich, Mark Jenkinson, Bartlomiej Papiez. Sir Michael Brady, and Julia Schn- abel. Towards realtime multimodal fusion for image-guided interventions using self-similarities. In Medical Image Computing and Computer-Assistedd Citerveentio-MMMCCA 2013: 16th International Conference, Nagoya, Japan, September 22-26, 2013, Proceedings, Part I 16, pp. 187-194. Springer, 2013.
Alessa Hering, Lasse Hansen, Tony CW Mok, Albert Chung, Hanna Siebert, Stephanie Häger, Annkristin Lange, Sven Kuckertz, et al. Learn2reg: comprehensive multi-task medical im- age registration challenge, dataset and evaluation in the era of deep learning. arXiv preprint 1XX1:21122004899 2021.
Malte Hoffmann, Benjamin Billot, Douglas N Greve, Juan Eugenio Iglesias, Bruce Fischl, and Adrian Dalca. Synthmorph: learning contrast-invariant registration without acquired images. IEEE transactions medical imaging, 41(3):543-558 2021.
Andrew Hoopes, Malte Hoffmann, Bruce Fischl, John Guttag, and Adrian V Dalca. Hypermorph: Amortized hyperparameter learning for image registration. In International Conference on Infor- mation Processing in Medical Imaging, pp. 3-17. Springer, 2021.
Andrew Hoopes, Malte Hoffmann, Douglas N Greve, Bruce Fischl, John Guttag, and Adrian V Dalca. Learning the effect of registration hyperparameters with hypermorph. The journal of machine learning for biomedical imaging, 1, 2022a.
Andrew Hoopes, Jocelyn S Mora, Adrian V Dalca, Bruce Fischl, and Malte Hoffmann. Synthstrip: skull-stripping for any brain image. Neurolmage, 360:119477 2022b.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. 2021.
Juan E Iglesias, Benjamin Billot, Yaël Balbastre, Colin Magdamo, Steven E Arolld, Sudeshna Das, Brian L Edlow, Daniel C Alexander, Polina Golland, and Bruce Fischl. Synthsr: A public ai tool to turn heterogeneous clinical brain scans into high-resolution t1-weighted images for 3d morphometry. Science advances, (5))eadd3607, 2023.
Juan Fugenio Iglesias. A ready-to-use machine learning tool for symmetric multi-modality registration of brain mri. Scientific Reports, 13(1):6657, 2023.
Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: self-configuring method for deep learning- based biomedical image segmentation. Nature methods, 18(2):203-211, 2021.
13

---

Published as a conference paper at ICLR 2025
Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier- Hein, and Paul F Jaeger. nnu-net revisited: A call for rigorous validation in medical image segmentation. arXiv preprint arXiv:2404.09556. 2024.
Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng. Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal orgti benchmark for versatile medical image segmentation. Advances in Neural Information Processing Systems, 35:36722-36732, 2022.
Jue Jiang, Neelam Tyagi, Kathryn Tringale, Christopher Crane, and Harini Veeraraghavan. Self- supervised 3d anatomy segmentation using self-distilled masked image transformer (smit). In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 556-566. Springer, 2022.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-81 19, 2020.
Ali Emre Kavur, M. Alper Selver, Oguz Dicle, Mustafa Barış, and N. Sinem Gezer. CHAOS Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data. Zenodo, April 2019. doi: 10.528//zemodo.3362844 URL https: /dii. org/10. 5281/ 2ewd//eeeoo.ooo 3362844.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 14Xiv:1412.6980 2014.
Markus Krönke, Christine Eilers, Desislava Dimova, Melanie Köhler, Gabriel Buschner, Lilit Schweiger, Lemonia Konstantinidou, Marcus Makowski, James Nagarajah, Nassir Navab, et al. Tracked 3d ultrasound and deep neural network-based thyroid segmentation reduce interobserver variability in thyroid volumetry. Plos one, 17(7):e0268550, 2022.
Pamela J LaMontagne, Tammic LS Benzinger, John C Morris, Sarah Keefe, Russ Hornbeck, Chengjie Xiong, Elizabeth Grant, Jason Hassenstab, Krista Moulder, Andrei G Vlassenko, al. Oasis-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. MedRxiv, pp. 2019-12, 12, 2019.
Ann Lee, David Mumford, and Jinggang Huang. Occlusion models for natural images: statistical study of scale-invariant dead leaves model. International Journal of Computer Vision, 41:35-59,, 2001.
Hyeon Woo Lee, Mert R Sabuncu, and Adrian V Dalca. Few labeled atlases are necessary for deep-learning- segmentation. arXiv preprint arXiv:1908. 04466, 2019.
Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Dreamteacher: Pretraining image backbones with deep generative models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 16698-16708. 2023.
Wenxuan Li. Chongyu Qu, Xiaoxi Chen, Pedro RAS Bassi, Yijia Shi, Yuxiang Lai, Qian Yu, Huimin Xue, Yixiong Chen, Xiaorui Lin, et al. Abdomenatlas: A large-scale, detailed-annotated, & multi-center dataset for efficient transfer learning and open algorithmic benchmarking. Medical Image Analysis, 97:103285, 2024a.
Wenxuan Li, Alan Yuille, and Zongwei Zhou. How well do supervised 3d models transfer to medical imaging tasks? In The Twelfth International Conference on Learning Representations, 2024b. URL https: /ttps://opeee /pppreeveeeeeeeeeeeeeeeeeeeeeeee net fooumm i d ZIPytk4.
Zudi Lin, Donglai Wei, Mariela D Petkova, Yuelong Wu, et al. Nuemm dataset: 3d neuronal nuclei instance segmentation at sub-cubic millimeter scale. In Medical Image Computing and Computer Assisted Intervention-MICCA 202/, pp. 164-174. Springer, 2021.
14

---

Published as a conference paper at ICLR 2025
M Linehan, R Gautam, S Kirk, Y Lee, C Roche, E Bonaccio, and R Jarosz. Radiology data from the cancer genome atlas cervical kidney renal papillary cell carcinoma KIRP collection. Cancer Imaging Archive, 2016.
Geert Litjens, Robert Toth, Wendy Van De Ven, Caroline Hoeks, Sjoerd Kerkstra, Bram Van Ginneken, Graham Vincent, Gwenael Guillard, Neil Birbeck, Jindang Zhang, et al. Evaluation of prostate segmentation algorithms for mri: the promise challenge. Medical image analysis, 18(2):359-33, 2014.
Jie Liu, Yixiao Zhang. Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 21152-21166 164, October 2023a.
Peirong Liu, Oula Puonti, Xiaoling Hu, Daniel C Alexander, and Juan Eugenio Iglesias. Brain-id: Learning robust feature representations for brain imaging. arXiv preprint arXiv:2311.66114, 2023b.
Peirong Liu, Oula Puonti, Annabel Sorby-Adams, William T Kimberly, and Juan E Iglesias. Pepsi: Pathology-cnhancee pulse-sequence-invaaiin represetttoosss for brain mri. arXiv preprint arXiv:2403.06227, 2024.
Calvin Luo. Understanding diffusion models: unified perspective. arXiv preprint arXiv:2208.11970 2022.
Jun Ma and Bo Wang. Segment anything in medical images. arXiv preprint arXiv:2304. 12300 2023.
David Mattes, David R Haynor, Hubert Vesselle, Thomas K Lewellyn, and William Eubank. Nonrigid multimodality image registration. In Medical imaging 2001: image processing, volume 4322, pp. 1609-1620. Spie, 2001.
Duy MH Nguyen, Hoang Nguyen, Nghiem Diep, Tan Ngoc Pham, Tri Cao, Binh Nguyen, Paul Swoboda, Nhat Ho. Shadi Albarqouni, Pengtao Xie, al. Lvm-med: Learning large-scale self- supervised vision models for medical imaging via second-order graph matching. Advances in Neural Information Processing Systems, 36, 2024.
Tony CW Mok, Zi Li, Yunhao Bai, Jianpeng Zhang, Wei Liu, Yan-Jie Zhou, Ke Yan, Dakai Jin, Yu Shi, Xiaoli Yin, et al. Modality-agnosti structural image representation learning for deformable multi-modality medical image registration. arXiv preprint arXiv:2402. 18933, 2024.
Kunio Nakamura, Jacqueline T Chen, Daniel Ontaneda, Robert Fox, and Bruce D Trapp. T1-/t2- weighted ratio differs in demyelinated cortex in multiple sclerosis. Annals of neurology, 82(4): 635-639, 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint wrXiv:2304.07193, 2023.
Marius Pachitariu and Carsen Stringer. Cellpose 2.0: how to train your own model. Nature Methods, pp. 1-8, 2022.
Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Zhu. Contrastive learning for unpaired image-to-image translation. European Conference on Computer Vision pp. 319-345. Springer, 2020.
Kelly Payette, Céline Steger, Roxane Licandro, Priscille de Dumast, Hongwei Bran Li, Matthew Barkovich, Liu Li. Maik Dannecker, Chen Chen, Cheng Ouyang, et al. Multi-center fetal brain tissue annotation (feta) challenge 2022 results. arXiv preprint 20t.:22022.046633 2024.
Fernando Pérez-García, Rachel Sparks, and Sébastien Ourselin. Torchio: python library for efficient loading, preprocessing, augmentation and patch- patch-based sampling of medical images in deep learning. Computer Methods and Programs in Biomedicine, pp. 106236, 2021. ISSN 0169-2607. doi:
URL https: www. www.:./.iinneeiee.......... com/ Sccience//rtticcl///i/////// /0266 02.
15

---

Published as a conference paper at ICLR 2025
Ken Perlin. An image synthesizer. ACM Siggraph Computer Graphics, 19(3)2287-996 1985.
Nicolas Piclawski, Elisabeth Wetzer, Johan Öfverstedt, Jiahao Lu, Carolina Wählby, Joakim Lindblad, and Natasa Sladoje. CoMIR: Contrastive multimodal image representation for registration. In Advances in Neural Information Processing Systems, 2020.
Huaqi Qiu, Chen Qin, Andreas Schuh, Kerstin Hammernik, and Daniel Rueckert. Learning dif- feomorphic and modality-invariian registration using b-splines. In Medical Imaging with Deep Learning, 2021.
Chongyu Qu, Tiezheng Zhang, Hualin Qiao, Yucheng Tang, Alan L Yuille, Zongwei Zhou, et al. Abdomenatlas-Sk: Annotating 8,000 volumes for multi-organ segmentation in three weeks. Advances in Neural Information Processing Systems, 36, 2024.
M. Ren, N. Dey, J. Fishbaugh, and G. Gerig. Segmentation renommallie deep feature modulation for unpaired image harmonization. IEEE Transactions on Medical Imaging, 2021.
Mengwei Ren, Neel Dey, Martin Styner, Kelly Botteron, and Guido Gerig. Local spatiotemporal representation learning for longitudinally-coniii information processing systems, 35: 13541-13556, 2022.
neuroimage analysis. Advances in neural
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pp- 234 241, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24574-4
Jo Schlemper, Seyed Sadegh Mohseni Salchi, Prantik Kundu, Carole Lazarus, Hadrien Dyvorne, Daniel Rueckert, and Michal Sofka. Nonuniform variational network: deep learning for accelerated nonuniform image reconstruction. In Medical Image Computing and Computer Assisted Intervention-MICCA. 2019 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part III 22, pp. 57-64. Springer, 2019.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 -57944, 2022.
Hanna Siebert, Lasse Hansen, and Mattias Heinrich. Fast 3d registration with accurate optimisation and little learning for learn2reg 2021. In International Conference on Medical Image Computing and Computer-Assisee Intervention, pp- 174-179. Springer, 2021.
Hanna Sicbert, Christoph Großbröhmer, Lasse Hansen, and Matias P Heinrich. Convexadam: Self- configuring dual-optimissttion- based 3d multitask medical image registration. IEEE Transactions on Medical Imaging, 2024.
Nalini M Singh, Juan Eugenio Iglesias, Elfar Adalsteinsson, Adrian V Dalca, and Polina Golland. Joint frequency and image space learning for mri reconstruction and analysis. The journal of machine learning for biomedical imaging, 2022, 2022.
Xinrui Song, Xuanang Xu, and Pingkum Yan. Dino-reg: General purpose image encoder for training- free multi-modal deformable medical image registration. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 608-617. Spriingrr 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Carsen Stringer and Marius Pachitariu. Transformers do not outperform cellpose. bioRxiv, pp. 2024-04 2024.
16

---

Published as a conference paper at ICLR 2025
Ryu Tadokoro, Ryosuke Yamada, Kodai Nakashima, Ryo Nakamura, and Hirokatsu Kataoka. Primi- tive geometry segment pre-training for 3d medical image segmentation. British Machine Vision Conference, 2023.
Saeid Asgari Taghanaki, Yefeng Zheng, Kevin Zhou, Bogdan Georgescu, Puneet Sharma, Daguang Xu, Dorin Comaniciu, and Ghassan Hamamnh. Combo loss: Handling input and output imbalance in multi-organ segmentation. Computerized Medical Imaging and Graphics, 75:24-33 33, 2019.
Yucheng Tang, Dong Yang. Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10730-20740, 2022.
Götz Thomalla, Bastian Cheng, Martin Ebinger, Qing Hao, Thomas Tourdias, Ona Wu, Jong Kim, Lorenz Breuer, Oliver C Singer, Steven Warach, et al. Dwi-flair mismatch for the identification of patients with acute ischaemic stroke within 4- h of symptom onset (pre-flair): a multicentre observational study. The Lancet Neurology, ():11) 986, 2011.
Lin Tian, Hastings Greer, Roland Kwitt, Francois-Xavies Vialard, Raul San Jose Estepar, Sylvain Bouix, Richard Rushmore, and Mare Niethammer. unigradicon: A foundation model for medical image registration. arXiv preprint arXhv:2403.05740 2024a.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024b.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Nicholas J Tustison, Philip A Cook, Andrew J Holbrook, Hans Johnson, John Muschelli, Gabriel A Devanyi, Jeffrey T Duda, Sandhitsu R Das, Nicholas C Cullen, Daniel L Gillen, et al. Antsx: A dynamic ecosystem for quantitative biological and medical imaging. medRxiv, 2020.
Jeya Maria Jose Valanarasu, Yucheng Tang, Dong Yang, Ziyue Xu, Can Zhao, Wenqi Li, Vishal M Patel, Bennett Allan Landman, Daguang Xu, Yufan He, et al. Disruptive autoencoders: Leveraging low-level features for 3d medical image pre-training. In Medical Imaging with Deep Learning. 2024.
Stéfan van der Walt, Johannes L. Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, Tony Yu, and the scikit-image contributors. scikit- image: image processing in Python. PeerJ, 2:e453, 6 2014. ISSN 2167-8359. doi: 10.7717/peerj. 453. URL https::t///..
doi. http://doooooo//////////o 7717 453.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2495-2555 -504,, 2021.
Jakob Wasserthal, Hlanns-Christee Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexan- der W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence, (5), 2023.
William M Wells III, Paul Viola, Hideki Atsumi, Shin Nakajima, and Ron Kikinis. Multi-modal volume registration by maximization of mutual information. Medical image analysis, 1(1):35-51, 1996.
Hallee Wong, Marianne Rakic, John Guttag, and Adrian Dalea. Scribbleprompt: Fast and flexible interactive segmentation for any medical image. arXiv preprint arXiv:2312.07338 2023.
Yutong Xie, Jianpeng Zhang, Yong Xia, and Qi Wu. Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier. In European Conference on Computer Vision, pp. 558-575. Springer, 2022.
17

---

Published as a conference paper at ICLR 2025
Chenyu You, Weicheng Dai, Yifei Min, Fenglin Liu, David Clifton, S Kevin Zhou, Lawrence Staib, and James Duncan. Rethinking semi-super medical image segmentation: A variance-reduucioo perspective. Advances in Neural Information Processing Systems, 36, 2024.
HeiShun Yu, Jonathan Scalera, Maria Khalid, Anne-Sophie Touret, Nicolas Bloch, Baojun Li, Muhammad M Qureshi, Jorge A Soto, and Stephan W Anderson. Texture analysis as radiomic marker for differentiating renal tumors. Abdominal Radiology. 42:2470-244 22788,47777.20 2017.
Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, and Prateek Prasanna. Self pre-training with masked autoencoders for medical image classification and segmentation, In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pp. 1-6. IEEE, 2023.
Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B Gotway, and Jianming Liang. Models genesis. Medical image analysis, 67:101840, 2021.
Xiahai Zhuang. Multivariate mixture model for myocardial segmentation combining multi-source images. IEEE transactions on pattern analysis and machine intelligence, 41(12))2933-2946, 2018.
Xiahai Zhuang, Lei Li. Christian Payer, Darko Štern, Martin Urschler, Mattias Heinrich, Julien Oster, Chunliang Wang, Orjan Smedby, Cheng Bian, et al. Evaluation of algorithms for multi- modality whole heart segmentation: an open-access grand challenge. Medical image analysis, 58: 101537, 2019.
18

---

Published as a conference paper at ICLR 2025
A
APPENDIX: ADDITIONAL RESULTS
A.1
ADDITIONAL SYNTHETIC DATA VISUALIZATIONS
Ours
Brains
smshapes
Figure 9: Randomly selected synthetic volume (center slice) visualizations sampled from our proposed data engine (top). For clarity, these samples have not had the online augmentations in Figure 12 applied to them. In the Brains and smshapes rows, we present samples corresponding our ablations where we replace our proposed 3D label ensemble generator with real 3D brain labels (middle) or synthetically generated labels with no biomedical priors (bottom), respectively, but keep appearance model unchanged.
19

---

Published as a conference paper at ICLR 2025
A.2 REGISTRATION REGULARIZATION WEIGHT GRID SEARCHES
Deformable registration optimization faces a trade-off between accuracy and warp field regularity, which often tackled by using various regularizers and hyperparameters (Hoopes et al., 2021). For a fair comparison with ConvexAdam, we separately tune both the original framework and our extension (ConvexAdam-Oune splits of both datasets (i.c., L2R-Abdomen MRCT and MM-WHS). These hyperparameters include: Adam (Kingma & Ba, 2014) optimization grid spacing: {1,2), 2}, the warp smoothness penalty X: (0.25, 0.5, 0.75. ... 2.5), grid spacing: {2, 32...... 6) and di sp hw: {1,2,... 55.. All hyperparameters are tuned such that post-registration volume overlap (Dice) maximized while maintaining deformation folds below 0.5%.
via grid search over four hyperparameters on the validation
In Table 4, we summarize the Dice and folding statistics over the validation set grid searches. The reported means are averaged across subjects and hyperparameter configurations while standard deviations indicate inter-configuration spread. Using our network features (ConvexAdam-On leads to substantially better performance and lower sensitivity to hyperparameter settings. This is corroborated Fig. 10, where for different settings of along the x-axis, we visualize Dice and folding voxel percentages for each hyperparameter configuration. We again find better performance at lower folding percentages while maintaining lower sensitivity to hyperparameter settings.
Table 4: Registration validation set grid search summary statistics (mean + std.). Here, the means are computed over all subjects and all hyperparameter configurations and the standard deviations correspond to the spread over all hyperparameter configurations.
L.2o-mn MRCT
MM-WHS
Method
Dice (t)
Folds% (1)
Dice (1)
Folds% ())
ConvexAdam Adam-Ours
0.863 0.016
0.693
1.423
0.661 0.030
0.496 + 1.249
ConvexAdam
0.806 + 0.038
2.269 + 3.292
0.652 + 0.023
1.715 + 3.572
Dice
Folds %
0.50
Corvexidom Carvesidam
Corwexedamm
Learn2Reg Abdomen MRCT
ConvexAsaem
0.70
0.65
0.75
1.0
1.0
1.75
2.0
Corvexidoon Ours
Opn Deskdwe
12
MM MMMWWH
0.70
0.75
1.0
1.25
1.25
2.25
2.5
0.25
0.5
0.75
1.0
1.25
1.75
2.0
2.25
Figure 10: Registration validation set grid search sweep statistics. Here, each point contained in boxplot is the average Dice for particular hyperparameter configuration, given fixed
20

---

Published as a conference paper at ICLR 2025
A.3
ADDITIONAL REPRESENTATION VISUALIZATIONS
Input volumes
Channel-wise Oulpul Representations
Pretrained Our Label Ensembles
Temperature 0.07
Verying the temperattue (7 the contrastive loss
Pretrained Our Label Eireebbles
Temperature 0,20
Pretrained Our Label Eireembles
Temperature = 0.33
Varying labels used for appearanace model synthesize images
Pretrained wi Adult Brain Labels
Temperature = 0.33
Pretrained w/ smshopps Labels
Temperature = 0.33
Figure 11: Companion figure to Fig. 7: Feature visualizations for varying pretraining configura- tions. Here, we visualize channel-wise output representations from our pretrained network for the two volumes at the top left for the first six channels. First, varying the temperature hyperparameter in the contrastive loss can lead to substantial aliasing (grouped rows 1-3). Then, changing our label ensemble synthesis model for other label sources (grouped rows 3-5) shows a loss in stability to nuisance variation. Our proposed model in grouped row 3 achieves both interpretable and stable representations on highly challenging volume pairs.
21

---

Published as a conference paper at ICLR 2025
Table 5: Experiments comparing the multitask capabilities of the second-best segmentation model in Table 1 (PrimGeoSeg) when matched for parameters and network architecture, as measured by subject-averaged Dice coefficients and their corresponding bootstrapped std. deviations.
Few-shol Segmentation
Registration w/ ANTs
Method
Architecture
Params.
MSD-Heart
1.2RAb-MRI
FrTA
AMOS-CT
WUFeraa
1.2RAb
MMWHS
PrimGeoSeg
SwiLLNETR
.87.01)
84(.05)
79603)
.6(,01)
.66.02)
.46,09)
.10.02)
PrinGooSoog
UNet UNet
5.9M
.8(.02)
84(.0)
796,03)
561.01)
.71.02)
.36,06)
48(.02)
Ours
5,9M
.89,01)
86(.05)
.80.03)
.61.01)
.76.02)
.70,09)
.63.02)
A.4 NETWORK PARAMETER COUNT EFFECTS
Our experiments in Section 4.2 compare our pretrained U-Net with publicly released foundation models that have significantly higher parameter counts. To investigate the impact of network size, we use the same U-Net architecture as in our proposed model and retrain the second-highest average ranking method, PrimGeoSeg (Tadokoro al., 2023), reducing its parameter count from 67.2M to 5.9M but matching all other training details to their code repository'. We then use it as feature extractor for multi-modan modlltt registration with the ANTs solver (Tustison et al., 2020) and also finetune it for few-shot segmentation, as in our experiments in the main text.
As shown in Table matching the parameters of PrimGeoSeg results in performance drops across 3 out of few-shot segmentation datasets, well in both registration datasets. Notably, while PrimGeoSeg had originally outperformed our method on the AMOS-CT few-shot segmentation experiment the main paper, its performance drops below ours by 5 mean Dice points when matched in parameter count, suggesting that the performance gap is function of network size on that dataset.
A.5 NEGATIVE RESULTS
While our proposed model consistently achieves state-of-the-art performance across several segmentation and registration benchmarks. it shows interesting trends on the preprocessed neurite-casis (Hoopes et al., 2022a) TIw MRI neuroimage segmentation dataset, which is derived from the larger OASIS (LaMontagne et al., 2019) dataset. For this few-shot segmentation experiment, we train on 160 erops, with a batch size of 3, and finetune on one annotated subject to segment the 35 classes provided by the dataset. In Table 6, we find that proposed pretrained network does not improve over random initialization for this particular dataset.
Table 6: Few-shot 3D segmentation results on the neur te-oasis dataset (Hoopes et al., 2022a) reported the mean Dice coefficient and bootstrapped std. deviation.
RandInit ModelsGenesis
MedicalNet PrimGeoSeg
SMIT
DisruptiveAE
Ours
.82(.01)
.8(,01)
.75(.01)
.84.01)
.8(.00)
80(.01)
.82(.1))
This result is consistent with the literature on the limited benefits of representation learning for few- shot adult neuroimage segmentation on OASIS in terms of Dice coefficient gains, as also reported in Ren (2022). These trends may stem from the relatively high resolution, tissue contrast, and low inter-subject variability in OASIS. Further supporting this, prior work (Lee et al., 2019) has also shown that training adult neuroimage segmentation models from scratch on very small datasets can yield competitive results. Of the 8 remaining segmentation and registration tasks, our model either achieves the best (7 of tasks) or second-best (1 of tasks) performance, demonstrating its overall robustness.
'https://githube http
22

---

Published as a conference paper at ICLR 2025
A.6 ADDITIONAL RESULTS ON ULTRASOUND AND MICROSCOPY DATASETS
Table 7: Few-shot segmentation Dice results on 3D ultrasound (SegThy) and 3D microscopy (NucMM-M) datasets, reported mean Dice score with bootstrapped standard deviations. The best- test-performin method is in bold and the runner-up is underlined.
Method
SegThy
NucMM-M
Finetuning amount
2 subjects
volume
Number of classes
3
Random Init UNet
82(.03)
87(.03)
Transfer Learning
.8((02)
88(.02)
Models Genesis
.78(.02)
83(.02)
MedicalNet
.77(.03)
88(.03)
PrimGeoSeg
.78(.03)
88(.03)
SMIT
.78(.03)
91(.01)
Disruptive AE
.74(02)
88(.01)
Ours
.84(.2)
89(.02)
The imaging modalities used in the datasets utilized for few-shot segmentation in Table consist of CT and various MRI sequences. We extend evaluation here include two additional datasets representing distinct volumetric biomedical imaging modalities: 3D ultrasound and 3D microscopy.
For ultrasound, we use the SegThy dataset (Krönke et al., 2022) which provides ultrasound scans from 28 healthy volunteers imaging the left and right side of the neck for each subject and labels for the thyroid, carotid artery, and jugular vein. We resample all volumes to a common spacing of 0.24mm² and use two subjects (i.e., four left/right volumes) for few-shot finetuning. We use subjects for validation and 6 subjects for held-out testing. For microscopy, we use the NucMM-M (L.in et al., 2021) dataset of nuclei in the mouse visual cortex at 0.480 x 0.510 x 0.510pmm² resolution. As NucMM-M M is an instance segmentation dataset, we repurpose it for semantic segmentation by binarizing the instance labels for consistency with other experiments. NucMM-M provides 8 annotated volumes, of which we use I for finetuning, for validation, and 4 for held-out testing.
In Table 7, for SegThy, the proposed model improves upon all baseline pretrained models by clear margin, demonstrating our model's usefulness to the unseen domain of 3D ultrasound. For NucMM-M, achieve 2nd place performance to SMIT, albeit with low inter-method variability. However, as in Tables 1, and 7, our method consistently outperforms SMIT across all other seven segmentation tasks and both registration tasks, highlighting its generalizability.
A.7 ADDITIONAL COMPARISONS WITH 2D DINOV2
Table 8: Few-shot segmentation results (mean Dice score with bootstrapped standard deviations) comparing our method to slicewise 2D) finetuning of DINOv2 (Oquab et al., 2023) finetuned with LoRA (Hu et al., 2021) and trainable FPN decoder head and randomly initialized 3D UNet.
MSD-Heart PROMISE12 L2RAb-MRI
FeTA
AMOS-CT
WUFetal
2D DINOv2 w/ LoRA, FPN
.78 (.01)
.72 (.02)
.72 (.07)
.61 (.04)
(.01)
.43 (.02)
3D UNet (Randomly Init.)
.85 (.01)
.80 (.02)
.85 (.06)
.78 (.03)
(,01)
.73 (.02)
3D Ours
.89(01)
.85(01))
.86(06)
.80(03)
.61(.11)
.76(.02)
While our primary segmentation baselines are 3D foundation models specifically trained for radiology, recent work (Cekmeceli et al., 2024; Song et al., 2024) demonstrates that 2D natural vision foundation models, such as DINOv2 (Oquab et al., 2023), can be used for medical image analysis as well. In particular, concurrent work by Cekmeceli et al. (2024) shows that DINOv2 can be adapted for medical image segmentation in a fully supervised, slice-by-slice setting. To this end, we additionally benchmark against adapting DINOv2 for few-shot segmentation across all datasets in Table 1.
For DINOv2, we use the pretrained ViT-L/14 distilled (with registers) encoder. The encoder outputs an image embedding and we adapt it for 2D multi-label segmentation by attaching trainable feature pyramid network (FPN) decoder head. We also train the encoder DINOv2 weights using low-rank adaptation (Hu et al., 2021) for efficiency. Since the encoder expects 3-channel inputs,
23

---

Published as a conference paper at ICLR 2025
Table 9: Few-shot segmentation results (mean Dice score with bootstrapped standard deviations)
comparing our method with finetuning an abdominal CT pretrained segmentation model (SuPreM).
Methods
MeD-Heart
PROMISE12
L2RAb-MRI
FcTA
WUFetal
SuPreM
.85 (.02)
.75 (.02)
.84 (.05)
.79 (.03)
.71 (.02)
Randomly Initialized UNet
.85 (.01)
.80 (.02)
.85 (.06)
.78 (.03)
.73 (.02)
Ours
.89 (.01)
.85 (.01)
.86 (.06)
.80 (.03)
.76 (.02)
single-channel medical images are repeated across channels. All other training details (e.g., training losses, iterations, learning rates) are matched to our 3D segmentation setup described in Appendix B.5 and we pick the weights with the best validation set performance to compute the test set scores below.
Table shows that while the 2D foundation model can be finetuned slice-by-slice for medical image segmentation, 3D Noomedically-fouusee approach consistently outperforms it for few-shot segmentation across all six datasets and distinct application domains considered. We also find that training well-tuned 3D UNet from random initialization generally outperforms slicewise application of a 2D natural image foundation model finetuned on that medical dataset, highlighting the need for natively 3D generalist approaches such ours. This finding aligns with other works benchmarking 2D vs. 3D self-supervised segmentation approaches as well, as in Ren et al. (2022).
A.8 ADDITIONAL COMPARISONS WITH A ABDOMINAL CT FOUNDATION MODEL TRAINED ON LARGE-SCALE DATA
In parallel to our work, large-scale annotated collection of 9.262 abdominal CT volumes was released (Li et al., 2024a) and the basis for high perfforrror segmentation foundation models for abdominal CT (Li al., 2024b). While these models perform well within their domain, it is unclear whether their pretrained weights transfer to new and unrelated biomedical domains and imaging modalities, such as fetal MRI. In contrast, method, trained highly variable synthetic data, does generalize consistently across diverse, unseen radiological domains and tasks.
To directly evaluate the transferability of abdominal CT foundation models to broader biomedical applications, we finetuned the SuPreM UNet (Li et al., 2024b), pretrained on Abdomen on all non-abdominal CT datasets considered in the main paper. All splits, augmentations, and other training details are consistent with the main body of the paper.
Table 9 presents few-shot segmentation results for SuPreM, our model, and randomly initialized UNet (with a matched architecture to ours), all finetuned on out-of-distribun datasets. SuPreM does not transfer well to new imaging modalities and biomedical applications. In contrast, our representations, trained only on highly variable synthetic data, generalize consistently.
We note that future work, our framework can also be trained with supplemental real volumes and speculate that training on a mixture of synthetic and surge-scale real datasets will have synergistic benefits, potentially yielding both high in-domain and out-of-domain performance.
24

---

Published as a conference paper at ICLR 2025
B APPENDIX: ADDITIONAL IMPLEMENTATION DETAILS
B.1 DATA ENGINE DETAILS
Our data engine in Fig. 2 A & B has several components. Here, describe low-level implementation details. We note that we make extensive use the MONAI (Cardoso et al., 2022), TorchIO (Pérez- García et al., 2021), and scikit-image (van der Walt al., 2014) libraries for both label and volume synthesis.
B.1.1 LABEL SYNTHESIS
The pseudocode in Algorithm summarizes the synthesis process for single label volume depicted in Fig.2A. This process assumes the availability of set Y of binary segmentation labels for different organs, which serve as templates. Specifically, these segmentations are taken from version of the publicly available TotalSegmentator dataset (Wasserthal et al., 2023) (CC BY 4.0 license). To incorporate binary labels with multiple connected components, we merge individual rib labels into a single binary label for all ribs and also similarly pool the individual vertebral labels.
In Algorithm 1, Pig and Penvelope refer to the probability of foreground masking and creating an the kernel width of the ball kernel used for
envelope around the foreground, respectively, morphological dilation and erosion. refers to a spatial deformation operator, and denotes the clement-wise multiplication. The Perlin deformations Ps used are taken from Hoffmann et al. (2021). Algorithm 1 3D synthetie label map L generation
Input:
dataset Y of binary templates
Output: Synthesized label map I.
1: Initialize L IR x28x 128x 128 with all zero entries
2: Sample N ~ {20, 40) templates T = (T,,...,TT, Tx} uniformly at random from Y
3: for i = 1, NN do
4: Center-crop and pad T; to (128, 128, 128)
5: Warp T; with random affine matrix A; s.t. T; + T; A;
translations and rotations are sampled from 21[-5, and U -1,, m] respectively scales and shears sampled from U150 0.5] and u[-0.5, 0.5). respectively
6: Assign + * Tj at spatial indices where T;
7:
end for
8: Median smooth L
9: if Pig > 1/3 where Pig L10 1] then
Foreground mask
10:
Sample binary sphere B 1228xx2 with radius and center where ~ 4448 } and ~~ [132, 96) independently along all axes
Warp with Perlin deformation Po t So Pa where o ~ 2([155 5] Foreground mask L as L t L S
12:
Increment L L + spatial indices where
14:
Penvelope > 0.5 where Penvelope ~ 2400 1] then
> Create envelope around foreground
Sample binary envelope 1122x1282 where
E = dilate(S, w) A (-(erode(S, w)) where ~ (2,
16:
Assign t- + at spatial indices where
17:
end if
18:
end if
B.1.2 VOLUME SYNTIESSIS
Given a label map L, we use it to conditionally sample two volumes/ccotrrastive views V1 and V/2 using appearance model that is summarized in Fig. 12. Specifically, the two intensity volumes are generated by sampling from two independent Gaussian mixture models conditioned on the label map then independently transformed by biomedical augmentation
L. These preliminary 3D images pipeline to form a contrastive pair of volumes. The term Zero background Fig. 12 refers to setting intensities in the volumes spatially coinciding with the background label to 0.
25

---

Published as a conference paper at ICLR 2025
(GMM Perlin, Label) Sample
Gibbs Ringing (o = 0.5)
Gaussian Blur (o = 0.5)
Bius Fiekl = 1.0)
K-space mask radiuus U|0,, 1/3]
(independencly per xyz axis) Ox U/0. 1:3]. oy C1o. 1/3] # - u[0, 1/3]
degree polynomnaa
Gaussian Sharpen (p 0.25)
coefficients U|C.0. 0.075]
amount Uf10, 30]
K-space Spike (p 0.2)
0 - UJ0.5, 1.0], 62 =
Intensity renge U|0.951, 1.10/]
Resolution Degracistion 1/3)
background 0.75;
(ils mear log- log-lntensiity)
resclution range L10.5,
Axis flp (p 1)
Gamma Adjustt =p= 0.5)
Gaussian Noise = 1/3)
Amnn Detormation (p ==000 1.0)
y 410.5, 2.0]
0.0, 0.25
per-axis translations Uf-16,
Gaussian Blur = 0.5)
K-spacc Motion = 1/3)
per-avás rotations L([rr/4, Tv/4
(independently per xyz axis) 0x U|D, 1/3), Sy U|O, 1/3] oz U10, 1/3)
(independently per-axis)
globel scaling U|-0.4, 0.4
translations Uf-10, 10]
rotations Uf-m/18, m/18
Min-max (0. 1] Intens ty Rescale
number of transtorms =
Synthesized (Image. Label)
Figure 12: Post-Gaussian mixture model volume augmentation pipeline used generate synthetic data for pretraining. Blue and red boxes refer to offline and online augmentations, respectively. All augmentations are applied to 128 x 128 x 128 volumes. p refers the pre-defined probability of applying an individual augmentation and all other hyperparameters mentioned are consistent with MONAI (Cardoso et al., 2022) conventions. All augmentations use the default library hyperparameters besides Gamma Adjustments, Gaussian Blurs, and Gibbs Ringing, where we reduce the range of corruption to avoid degenerate samples.
As the combined pipeline of our label and volume synthesizer is computationally intensive, generating data on the fly could potentially bottleneck training. To mitigate this, we sample 120,000 3D label volumes and their corresponding 240,000 contrastive volume pairs offline using the proposed model. During training, we further apply additional online augmentations using a smaller pipeline. The offline and online augmentations indicated in Fig. 12 by the blue and red boxes, respectively.
B.2 NETWORK ARCHITECTURES
We employ a widely used (Billot et al., 2023b; Ren et al., 2022) UNet architecture for pretraining, fine- tuning, and for use as a feature extractor in our baseline comparisons and ablations. The architecture described in Table 10. Each layer that contributes to the contrastive loss also includes individual MLP network that projects sampled spatial indices onto an embedding space. Specifically, the architecture of each MLP consists of two (128) BN ReLU blocks (where FCC (w) is fully connected layer of width w), followed by an FC (128) layer and an Lg-normalization sequence at the final layer.
B.3 PRETRAINING IMPLEMENTATION
We compute the contrastive loss in Eq. on pre-activation convolutional features extracted from layers 7, 9, 12, 15, 18, and 23 in Table 10. Model selection is performed by tracking the validation contrastive loss to pick best-performing checkpoint. We train using Adam (Kingma & Ba, 2014) with starting learning rate of 2 10-1 and step decay towards O every 120,000 iterations.
B.4 REGISTRATION EXPERIMENTS IMPLEMENTATION
Below, we first provide details regarding how the baselines were implemented in registration experiments and then describe how our network features were integrated with existing solvers. For volumes of arbitrary grid sizes, overlap ratio 0.8. Further, we use region-of-ing oegion-off-iterees interessst masks for fixed and moving volumes whenever registration method can use them.
sliding window inference with window size of 128 and an
26

---

Published as a conference paper at ICLR 2025
Table 10: U-Net architectural details. We the architecture from (Billot et al., 2023b; Ren et al., 2022).
refers to a sequence of 3D convolution with 3 x 3 x 3 kernels, batch normalization, and pointwise ReLU activations. ne is the channel width multiplier and n is the number of output channels. In our experiments, both n. and are set to 16.
Layer index
Layer contents
Conv-BN-ReL. Conv-BN-ReLUmna
2
Conv Bonv-BN-Relle (u
3
MaxPool(2), Conv-BN-Re10(22))
4
Conv-BN-Re2LU(hh)
MaxPool(2), Conv-BN-RRe100nn)
6
Conv-BNN-ReT1004n..
MaxPool(2),
Conv-BN-RReT.0000
8
Conv-BN-ReeIU(Sn))
MaxPool(2), Conv-BN--ReLU((6nn))
10
Conv-ENN-Re11J016nn.
11
Upsample 2x, Concatenate  withlayyee
12
Conv-BNN-Ree1J(16nn.
13
Conv-BN-RReJJ(16nn)
14
Upsample 2x, woncatenate with layer
15
Conv-BN-ReeL0(nn)
16
Conv-BN-RReI1(dn.)
17
Upsample 2x, Concatenate witthhaee layer
18
Conv-BN-Re1.0(2n. Conv-BN-Re11((n.)
19
20
Upsample 2x, Concatenate with layer
21
Conv-BN--ReLUOn.) Conv-BN--eLU(m.)) Conv-BN-ReLU(n)
22
23
B.4.1 BASELINE IMPLEMENTATION
SynthMorph-shapp
a domain-randomized diffeomorphic registration UNet trained synthetic volume pairs generated from label maps. It is optimized using a Dice loss subject to diffusion regularization and has no tunable hyperparameters at test time. We download its pretrained weights from the VoxelMorph library and use their Vensorflow-basse registration framework. Lastly, as their network is fully convolutional, we use the input volumes at their native resolution without resizing.
uni GradICON is an approximately diffeomorphic registration foundation model trained variety of datasets. We the binaries available their repository for our experiments. The off-the-shelf model does not have any hyperparameters at test-time and we use the default hyperparameters for the iterative variant (uniGradIcoN+TO)
ConvexAdam. ConvexAdam is a high-performance multimodality registration solver and we use the 67171f8 commit of the repository". Here we use the masked variant with default number of instance optimization iterations (80) and default hyperparameters of the MIND-SSC loss so as to use the same implementation MIND-SSC across experiments. All the remaining parameters are fine-tuned on the validation data, see Appendix A.2 for more details.
ANTs-MI. The ANTs-MI baseline was run with the following command using the ANTs library:
antsRegistration
27

---

Published as a conference paper at ICLR 2025
--verbose
dimeesiionaltty 3
--float 1
--output COOTPUTTFFO.DER/OOOee
OUTFUTT-QQDDRR//vveeedoo ume. n wwteeenii.gge nii.gzl,
--transform SyN[0.
-mmetric MI [f wum gz, gz, noving_volume.nis 1, 48, Random, 0. 0.666]
convergence 200x200x100
-srink-factors 3x2x1 \
smoothiing-sigmas 3x2x0vox
-- intterrelllaioon Linear \
--masks
moving_voluum__mase .rik mii.. .ziigz]]
where fixed_volume mii. and movingg_volumme
volume.nii are the input volumes to reg- ister and fixed volume mask mask gz and noving_volune, saces_masskk mii. gz are binary masks indicating non-zero / non-background regions.
fixed_volume
These settings correspond using a three-level registration pyramid with the SyN algorithm and the mutual information loss for 200, 200, and 100 iterations at each level with corresponding level-specific smoothing kernels.
B.4.2 MODIFICATIONS TO EXISTING REGISTRATION METHODS TO USE OUR NETWORK FEATURES
As our network produces 16 output channels, we modify existing registration solvers as below.
ANTs. To solve for warp between fixed and moving volume pair, define 16 different MSE-based loss functions with ANTs. Specifically, each loss estimates the dissimilarity between corresponding channel volumes produced by our network for the fixed and moving inputs. We also downscale each individual loss by tenth to trade off multiple data fidelity terms and regularization. The remaining modeling decisions and hyperparameters are identical the baseline and the following command:
antsRegistration verbose 1 \
dimensionality
-- --float 1
OUTFUTT_F..DRRRROoeddd ume. .nie.zz, --trasffomm SyN [0 15]
-output [OUTPUT_FOLDER// FOLDER/mOved_,
-connvereence 200x200x100
shrink-factors 3x2x1 \
smoothing--iigass 3x2x0vox
inteerpolation Linear \
--masks [fixed_volune wtt___ooooeeemsseese_eeee miw.gziii noving_volume_net wwt.mma..niii...
--metric MeanSquares [fixed_chl nii ni,
gz, moving_chl.nii. mzi. 0.1, 1, Random, 0. 666]
--metric MeanSquares [fixed_chiiee nii gz, moving_ch16.nne nii gz, 0.1, 1, Random, 666
ConvexAdam. ConvexAdam already operates on multichannel inputs by using handcrafted MIND- SSC features. We therefore concatenate our network features with their original features and additionally multiply the network features by 0.1 for stable optimization. We perform grid search over the same hyperparameters as the bascline ConvexAdam and the remaining modeling decisions to be consistent with it.
B.4.3 USING EXISTING 3D BIOMEDICAL SEGMENTATION FOUNDATION MODELS FOR REGISTRATION
In Section 4.3 and Table 2 of the main text, we demonstrate that current 3D biomedical segmentation models do not produce features that are directly usable by registration solvers such as ANTs. To elaborate,
was excluded from analysis it only provides pretrained encoder without a decoder. Model 1 sGenesis produces one output feature map for an image that we use as is. The SwinUNETR- architectuee based baselines (PrimGeoSeg, SMI and DAE) produce 48 output
28

---

Published as a conference paper at ICLR 2025
channels that would necessitate prohibitively high compute costs at full-resolution. However, based on validation set performance, we find no significant difference between using ANTs with all 48 features subset of 16 (every fourth channel), and thus use the latter for all results reported for the test set. The ANTs hyperparameters for all experiments are the same as in App. B.4.2, varying only in the number of input channels corresponding the output features produced by each method.
B.4.4 MM-WHS DATA PREPARATION
Prealignment. The public proportion of the MM-WHS dataset consists of 20 unpaired and annotated 3D CTs and MRIs of the heart, all from different subjects. The CTs are high-resolution CT angiograms with tight fields of view around the heart, whereas the MRIs often include the subject's s trunk. As our deformable registration baselines all assume affine pre-alignment, we align all the volumes to a common space. For accurate groupwise registration to this space, we formulate this affine atlas construction and registration problem (Avants et al., 2010)-
All CT volumes first clipped to [-450, 450] HU. We then arbitrarily select the first CT volume within the dataset (by subject ID) as an initial reference. We first resample it to a grid size of 160 160 128 1.142 1.142 x 1. 1.283mm resolution define an initial coordinate system. This resampled volume is then used as an initial target for affine atlas construction with the remaining CT volumes. We use the following ANTs command for groupwise affine alignment:
ant art.ahttttaaarccccceeeeeeeeeettr
-a 2 \
-d 3 \
-A 0 \
-o ((outputPath)T_
-g 0.2 \
-j 10 \
-n D \
O \
-1 4 \
2
-MI MI
1 I
-t Affine
-9 100x50
-f 4x2 A
-8 2x1
-b 1
-y D \
initial target_ct. .nii.. g2 pappu_cct++nni..1
Once affine CT atlas is estimated, then similarly register all MRI volumes to this CT atlas. In particular, due to the difficulty of intensity-based affine registration between the MRI and CT collections due domain and FOY shifts, we estimate these affine transformations the segmen- tations provided by the dataset and not the volumes themselves. Once all subject-to-athe affine transformations are estimated the segmentation volumes, they are used warp all of the intensity volumes into the desired common space. All deformable registration experiments in our paper use only the intensity volumes and use the segmentations only for evaluation.
Additional labels. MM-WHS provides manual annotations for seven structures including heart chambers and portions of arteries for its original use in segmentation benchmarking, However, there are several anatomical structures in these volumes beyond the original labels such as the spine. Therefore, to better repurpose this data for holistic and non-local multi-modality registration evaluation, annotate additional labels for the descending aorta and the spine. We use TotalSegmentator (Wasserthal et al., 2023) to segment these labels on the CT volumes and manually verify the results. For the MRI volumes, these new structures annotated by domain expert.
'ttpps: thub wwt.hhw...wwwww////////.a./ http://www..///aa////a//aaaaa/as ant vari atee a htt.l///mmeteeeeeeee.ee
29

---

Published as a conference paper at ICLR 2025
Table 11: Segmentation experimental dataset statistics. significantly differ from dataset to dataset.
All MRI modalities and sequences
WUFetal
PROMISE12
MSD-Heart
L2RAb-MRI
AMOS-CT CT
FeTA
Grid size
(112, 112, 80)
(320, 320, 24)
(320, 320, 115)
(192, 160, 192)
(512, 512, 115)
(256, 256, 256)
Original res. (mm²)
(3.0, 3.0,
(0.625, 0.625, 3.6)
L.25, 1.37)
2.0)
(0.68, 5.0)
(0.5, 0.5)
Training res, (mm²) "
(3.0, 3.0, 3.0)
(0.625, 0.625, 0.625)
(1.25, 1.25, 1.37)
(2.0, 2.0, 2.0)
(1.5. 1,5, 2.0)
(0.5, 0.5, 0.5)
Training crop size
80
96
96
128
96
128
Training batch size
4
4
Num. of labels
7
Modality
BOLD MRI
T2w MRI
DSSFP MRI
SPIR MII
CT
HASTE MRI
Finetuning vols.
3
2
1
3
3
Full supervision vols.
60
50
6
24
180
40
Validation vols.
15
20
10
20
Testing vols.
24
30
100
20
B.5 SEGMENTATION EXPERIMENTS IMPLEMENTATION
All image characteristics for each dataset summarized in Table B.5. The datasets and modeling decisions were chosen to maximize diversity between the various segmentation settings.
For the publicly available datasets, (re)split whatever annotated data is publicly available from the respective datasets. In particular, we resplit HeD-H FeTA, and L.2RAb-RR'S publicly available training sets obtain training, validation, and held-out testing data. For AMOS- CT,, we resplit the 200 volumes in the training set to obtain new training and validation splits and use their original and public 100 validation volumes held-out testing data. For PROMISEI2, 2, use the public splits, considering the test and livechal engetest splits to be its testing and validation splits, respectively.
PROMISE12 and AMOS CT are highly anisotropic in resolution and are thus resampled 0.625 0.625 x 0.625 mm and 1.5 x 1.5 x 2.0 mmn& resolution, respectively, with the latter resolution for AMOS-CT chosen to match the CT finetuning setting of the SwinUNETR baselines (Valanarasu et al., 2024). The CT intensities of AMOS -T were clipped to [-450, 450] HU and the spatial grid extents of MSD-Heart and (post-resampling) PROMISE12 were padded to have a minimum of 96 slices.
pretrained baselines use the crop sizes used for pretraining and finetuning their respective original papers for both consistency and due to their pretrained transformer backbones. All augmentations performed online and the MRI and CT dataset experiments in Table B.5 use the augmentations listed Table 12 top and bottom, respectively. We found all baseline models and model yielded better segmentation results when fully finetuned instead other popular evaluation strategies for pretrained models such as linear probing. Additionally, with gradual learning rate decay consistently improved results on validation splits. Therefore, when finetuning, train for 37, 37,500 iterations with batch size of four 3D crops using Adam with starting step size of 2 x 10 cosine decayed to O. Finally, we exclude the background label when reporting Dice statistics.
Our UNet baselines use the crop sizes listed in Table B.5 and our SwinUNETR--aann
found that finetuning for higher number of iterations
B.6 ALTERNATIVE LABEL SYNTHESIS MODELS
As in Sec. 4.3/label generation, we study the effect of our label ensemble synthesis model by replacing it with other label models used in biomedical image analysis. We detail implementation of these alternatives below and visualize them in App. A.I.
smshapes. For smshapes, follow the implementation of Hoffmann al. (2021) for label generation with two key exceptions. First, for the label synthesis model, they fix the number of labels to synthesize 26. For fair comparison with framework which varies the number of labels in each volume, we modify smshapes to sample the same variable number of shapes. Second, for the appearance model, match the hyperparameters of their GMM implementation to and also
30

---

Published as a conference paper at ICLR 2025
Table 12: Augmentations for segmentation fine-tuning experiments for MRI datasets (top table) and
CT datasets (bottom table). Hyperparameters correspond to MONAI conventions (Cardoso et al., 2022).
Prob.
MRI augmentation
Hyperparameters
1.0
Spatial crop
Crop size
0.33
Gaussian Noise
μ = 0.0, o = 0.1
0.33
Bias field
coefficients ~ 0.0005
0.33
Gibbs ringing
a~ 00, 0.33]
0.33
Gamma transform
7 [0, 4.5]
0.33
Gaussian blur
per-axis ~ 2(0 0.1
0.33
Gaussian sharpen
[0], 01 ~ 10 3.0). 02 1.0 rotation~ 41--//4,744) scale~ 4/0.8, 1.2], shear- U[2002,,0
1.0
Affine warp
(all per axis)
Prob.
CT augmentation
Hyperparameters
1.0
Spatial foreground crop
Crop size, foreground label weight of 0.5
0.33
Gaussian Noise
/ 0.0, 0.1
0.33
Gamma transform
7 ~ [.5)
0.33
Gaussian blur
per-axis 0.1
0.33
Gaussian sharpen
a~ [1,30 ~ 0, 3.0). . 02 ~ 1.0
1.0
Affine warp
rotation~ U1 [([-4/4,7/4] scale~ 2400.8, 1.2], shear- 0[-00220.2] Lunslation-441332222 32] (all per axis)
multiplicative Perlin noise such that the appearance models are now matched and only the source of labels varies.
Brains. As opposed to synthesizing labels, the Brai ns experiment uses real brain labels in manner similar (Billot et al., 2023b). This is done study the effect synthesized label ensembles with randomized positions versus real biomedical anatomical configurations. We use 492, 500, and 58 T1- weighted brain scans from ADNI, HCP, and IXI, respectively, and segment them with SynthSeg (Billot et al., 2023b) to obtain training label maps. Using these labels, we sample 120,000 label volumes with 240,000 contrastive views in our proposed model and match all other hyperparameters.
B.7 ALTERNATIVE PRETRAINING LOSSES
Denoising pretraining. For denoising pretraining, we maintain our data engine but pretrain the UNet to instead invert the intensity augmentations applied to the output of the initial Gaussian mixture model per sample, which is inspired by Iglesias al. (2023). The UNet has matched architecture to ours with an additional single convolutional layer mapping to a single-channel output volume. We train using the L1 loss denoising, match the optimization hyperparameters to our method, and select the checkpoint with the best validation L1 loss.
Removing label supervision. As our data engine provides exact label supervision for each voxel, use multi nolti-positive label-supera aube-superrvieed contrastive learning. However, several large-scale models pretrained with self-supervised objectives not using any label supervision. To benchmark against these approaches,
the self sele-supervised positive pair-only non-contrastive framework of Ren et al. (2021) with its losses applied to the same network layers as ours. Its variance and covariance loss weights are 0.01, the orthogonality weight is 100, and halve the initial learning rate for stable training on our data as opposed to real brains used in their work.
B.8 WUFETAL DATASET DETAILS
The in-house Whole Uterus Fetal (WUFetal) BOLD MRI dataset consists of 99 whole uterus volumes covering various pathologies, gestational ages, imaging artifacts, and the presence of twins. Due to this variability, this a highly challenging dataset for few-shot segmentation. These scans were acquired 3T Skyra Siemens scanner using multi-slice gradient echo EPT sequences 3mm
31

---

Published as a conference paper at ICLR 2025
isotropic resolution (TR [5-8] ms, TE [32-38] ms. x/2). All analyses performed retrospectively on anonymized data and are IRB-approved.
32


Reviews:
[{"review_id": "hCejq4vkvv", "reviewer": "Reviewer_8tqR", "summary": "The work proposes a new method for general (few-shot) medical image segmentation and registration. The method is based on generating realistic and varying synthetic data and training a generalist 3D segmentation and registration models on this data. The goal is to develop a method for generalizing to different imaging devices, medical procedures and conditions as well as different populations. \n\nThe data synthesis engine is capable to generate highly diverse samples and uses the totalsegmentattor dataset for shape templates. The “foundational” model is then pretrained using contrastive pretraining and finetuned in multi-modality registration and few-shot segmentation. \n\nIn experiments the authors demonstrate excellent performance in both downstream tasks.", "questions": "Please see weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tInnovative approach with a highly powerful general synthetic data engine, I think this work adds a lot to the discussion on “foundational” models in medicine. \n\n-\tThe method requires a comparatively small number of trainable hyperparameters compared to existing models while achieving a higher accuracy. This in my opinion is an important contribution towards sustainable models. \n\n-\tThe authors provide the source code which is a major plus for reproducibility and discuss reproducibility aspects. \n\n-\tThe authors evaluate the effect of different pretraining strategies in reasonable ablation studies. \n\n-\tThe authors honestly show negative results in the appendix.", "weaknesses": "-\tConsidered datasets. If I am not missing any details the authors mostly evaluate their method on CT and MRI images. They even consider an MRI image as out of distribution. I think experimentation and ablation on more difficult and diverse datasets, such as 3D microscopy or Ultrasound, even if the results are not all positive would add to the discussion and validate the claim of a “general volumetric biomedical foundation model”\n\n-\tFraming of contribution. I would prefer if the authors tone down their wording about the model and clearly point out that it is a model for radiology or CT and MRI dataset and not a “general volumetric biomedical foundation model”.\n\n-\tHyperparameter selection: What range of hyperparameters was tested, and how much time or resources were spent on tuning? How were the hyperparameters for the four baseline methods chosen? Especially for fine-tuning the baselines on your datasets, which I assume is done? Clearly describing the hyperparameter search is important for reproducibility. Please correct me if I was missing such details from the main manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "68lvkMjFas", "reviewer": "Reviewer_Emp1", "summary": "Authors present a method to generate highly variable synthetic imaging data which is then used to pre-train a 3D network using contrastive learning. The data generation method consists of drawing samples from a semantically labelled repository of biomedical shape templates to randomly populate an empty 3D volume. The volume is then deformed. Empty space and organ 'envelopes' are also simulated. To simulate different modalities and protocols, random intensity transformation is applied to the deformed 3D volume to yield 2 images. Typical imaging artifacts such a sbias field and blurring are simulated through random augmentations.The two images are fed into a UNet, and contrastive pre-training is performed on features at each decoder layer. An anchor point is chosen in one of the images, and all voxels of that label in both images are considered positive, and everything else negative pairs. The network yields features that can be used to finetune on other modalities and tasks. Importantly, the representations are modality-agnostic and anatomy-agnostic.", "questions": "* Authors compare their randomized shape template-based synthetic data engine to one that uses data with no biomedical priors and one using brain regions. Can Authors elaborate more on the intuiton for why their randomly deformed template shapes are so effective? Is there some point at which the extent of the deformation causes the representations to be less useful?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper is very well written - it lays out the prior work and puts the contirbute in context.\n* The approach yields representations that are both modality-agnostic and task-agnostic while removing the need for dataset-specific and anatomy-specific pre-training.\n* Authors present results of several downstream tasks using their features including multi-modality registration image registration and few-shot segmentation on which their method outperform the others compared.\n* Authors perform ablation studies on the various components of their pipeilne.\n* The Authors present extensive visualization and quantitative results in their main text, and supplementary material. Algorithms and parameters are clearly presented too which allows for further scrutiny and improved reproducability.\n* Authors are aware of the limitations of their approach and include these in the paper.", "weaknesses": "* The segmentation task performed using the Authors features may yield better results than the other methods that are compared, however the result still misses significant portions of the anatomical regions they aim to segment. The features require further adjustment and extensive fine-tuning to be useful in diagnosis and treatment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ASDwKTL3X", "reviewer": "Reviewer_8WY8", "summary": "This work proposes a pre-training approach for downstream tasks related to fine-grained volumetric medical data: image registration and semantic segmentation. The authors propose to learn appearance invariance and shapes of human anatomy through synthetic dense pixel volumes. In this process, 3D volumes are synthesized by randomly recomposing 3D anatomical shapes and assigning multiple sets of random pixel values, in together with synthetic noises and deformations. Pairs of synthetic volumes are used for multi-scale contrastive learning. The proposed approach demonstrates improved image registration and low-shot image segmentation results compared to some previous works. Detailed ablation studies on the pre-training configurations toward downstream performances are conducted.", "questions": "The proposed workflow involves many hyper-parameters (Figure 12) controlling the properties of generated synthetic volumes -- what is the rule of thumb for choosing them?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The overall methodology is straightforward and easy to understand. It echoes with the classical computer vision concept of invariance learning in the deep neural network era (although learned through a data-driven approach).\n\nImproved image registration results on two public image registration benchmarks and image segmentation performance on six image segmentation datasets are shown, compared to those of some existing works. \n\nThe paper is well-written with sufficient clarity. The illustrations are self-explanatory. Readers will enjoy reading it.", "weaknesses": "Technical novelty: The core idea behind the approach is to leverage data-driven approach to learn invariance to pixel values through paired synthetic shapes and different pixel values, and to learn semantic-independent shape representation through random geometric (real-world and pure synthetic) shapes – both key ideas come from the well-established SynthMorph (random Ising shapes with random intensity + synthetic deformation for training registration networks. Hoffmann et al.) and SynthSeg (GMM-like pixel value model for repainting brain structures. Billot et al.) Despite leveraging more anatomical shapes beyond brain structures and applied to a contrastive framework, the essence remains unchanged.  \n\nMany medical decisions are made not only on shape but also on subtle textures, for example, differentiating subtypes of tumors/lesions – toward which the proposed over-simplified appearance model by nature falls short. More sophisticated texture models need to be carefully studied beyond this manuscript.\n\nFor the same reason, high-level global semantic information such as relative locations between anatomical structures cannot be learned due to the nature of this approach. \n\nReal-world value: Given the increasing number of large-scale publicly accessible volumetric image datasets such as CT-RATE (Hamamci et al.), Totalsegmenter (Wasserthal et al.), and AbdomenAtlas (Li et al.), and the derived 3D foundation models, the real-world application of the proposed framework is unclear. Some of these large-scale public datasets come with fine-grain pixel-wise labels and associated radiological reports which provide additional supervision signals and text alignment potentials. The claimed generalization capability can be learned from multi-site large real-world datasets as well, through the intrinsic heterogeneity of big data and possibly through intense data augmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NmPeyXL6j9", "reviewer": "Reviewer_voNv", "summary": "The paper proposes training a backbone that generalizes across different datasets using synthetically generated dataset. The proposed pre-training strategy has 3 main steps: 1) Given a large datasets of 104 annotated organs, randomly sample anatomies, deform them and create a volume by ensembling these anatomies, 2) add noise and other augmentations to the volumes that are sampled in the previous step to simulate realistic looking synthetic medical images from labels, 3) train a U-Net with a contrastive objective by sampling two volumes that share the same 3D semantic layout but differ in appearance, treating corresponding features at different encoder levels as positives and all others as negatives. The pre-trained backbone is validated on two different tasks: 3D registration and 3D few-shot segmentation; using multiple datasets.  The results show the effectiveness of the proposed backbone in the experiments compared to existing methods.", "questions": "1- What are the exact steps of the \"label ensemble model\" described in Section 3? Please write elaborate description of these steps.\n2- Why do the generated images not look like real medical images? Is it because the deformation is too large? Why such \"unrealistic\" looking images are preferred rather than more realistic ones obtained with smaller deformation?\n3- How does the quality of the representations obtained by the proposed backbone compares with SoTA foundational models such as DinoV2 or SAM2?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Foundational models are showing promising performance lately; however, we lack of a 3D model that work across different modalities in medical imaging. The paper proposes a solution to this important problem using domain randomisation and contrastive learning.\n- The paper contain experiments on multiple datasets both for registration and few shot segmentation, and the results demonstrate the potential of the method.\n- The idea of combining the ideas of domain randomisation and local contrastive learning to train a generic 3D backbone is quite interesting and, to my knowledge, is novel.", "weaknesses": "- One issue I see in the paper is the convoluted description of the data engine step, especially the part creating the label ensemble model in section 3. I understand that this step is mainly based on the domain randomisation idea proposed in the literature. However, it is not really clear to me the steps between 201-207, especially the parts multiplying the masks with a randomly deformed sphere and randomly encasing half of the foreground-masked volumes.\n\n- The images generated in the data engine step do not seem like as real medical images. Do they look like this because the deformation is too large? It is not clear why one would prefer training the model using such unrealistic images.\n\n- The paper does not discuss the recent foundational models that show better generalization performance on many medical image datasets [1]. The downstream task performance of the representations obtained from the proposed backbone should be compared with the those obtained by the representations of a foundational model (e.g. DinoV2 [2]). For example, [3] is a recent paper that uses DinoV2 features for registration; but the same applies for the segmentation experiments. One can use the DinoV2 features for segmentation.\n\n[1] Cekmeceli et al. \"Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?\"\n[2] Oquab et al. \"DINOv2: Learning Robust Visual Features without Supervision\"\n[3] Song et al. \"DINO-Reg: General Purpose Image Encoder for Training-Free Multi-modal Deformable Medical Image Registration\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)