Ms-Multi-Spine challenge proceedings

Benoit Combès, Anne Kerbrat, Romain Casey, François Cotton, Michael
Kain, Sorina Camarasu-Pop, Michel Dojat

<figure><img src="image_1.png" /></figure>

To cite this version:

Benoit Combès, Anne Kerbrat, Romain Casey, François Cotton, Michael Kain, et al. Ms-Multi-Spine challenge proceedings. MS-MULTISPINE 2025 - Miccai Challenge on MS Lesions Segmentation in Spinal Cord MRIs, Oct 2025, Online event, France. pp.1-35, 2025. inserm-05317094

HAL Id: inserm-05317094

https://inserm.hal.science/inserm-05317094v1

Submitted on 16 Oct 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
## Ms-Multi-Spine challenge proceedings:

### "Multiple Sclerosis Spinal Cord Lesion Detection from MultiSequence MRIs"

https://portal.fli-iam.irisa.fr/MS-Multi-Spine/

<figure><img src="image_3.png" /></figure>

## Organization team:

Benoit Combès, Anne Kerbrat, Romain Casey, François Cotton, Michael Kain, Sorina Pop, Michel Dojat.
©ms-multi-spine 2025, MICCAI challenge on multiple sclerosis spinal cord lesions segmentation.

The papers included in this proceedings book were part of the technical challenge cited on the cover. The papers were selected and reviewer. The papers published in this proceedings book reflect the work and thoughts of the corresponding authors and are published herein as submitted with minor editorial revisions. Neither the editors, nor the challenge organizers can accept any legal responsibility for any errors or omissions that may be made. Please use the following format to cite materials from this proceedings book:

<Authors>, <Paper Title>, In: Proceedings of the ms-multi-spine challenge on Spinal Cord Lesion Detection from MultiSequence MRIs | MICCAI-ms-multi-spine, Combès, Kerbrat, Casey, Cotton, Kain, Pop, Dojat. (Eds), pp. <Page Numbers>, 2025.
# Preface

This proceedings book gathers the methodological papers accompanying pipelines submitted for the ms-multi-spine-challenge<fnref n="1" />, that took place in the context of Miccai 2025 conference. This challenge results from the joint motivation of the OFSEP<fnref n="2" /> (French registry on multiple sclerosis aiming at gathering, for research purposes, imaging data, clinical data and biological samples from the French population of multiple sclerosis subjects), FLI<fnref n="3" /> (France Life Imaging, devoted to setup a national distributed e-infrastructure to manage and process medical imaging data) and Empenn research team. These particular efforts were directed towards bringing attention to the spinal cord multiple sclerosis lesion segmentation and to its specific methodological setting. Indeed, in clinical practice, it is highly recommended to acquire at least two sequences among a set of available ones for the detection of spinal cords, without specific guidelines to date. In practice, depending on the center and context, any combination of existing MR sequences can be provided. This challenge therefore represents a concrete and paradigmatic case of missing modalities setting where, depending on the case, some modalities may be missing both at inference or training time. To the best of our knowledge, such clinical datasets are still rarely available in medical imaging. More generally, to date, the medical imaging community concentrated its efforts toward the detection/segmentation of the lesions in brain MRI but spinal cord lesions remain a topic much less studied.

In this challenge, we provided a set of 100 segmented cases with different combinations of sequences among sagittal T2w, sagittal PSIR, sagittal STIR and 3d MPRAGE to the participants, that were asked to propose a segmentation method being able to deal with any of the following combinations (sagittal T2w, sagittal PSIR), (sagittal T2w, sagittal STIR), (sagittal T2w, 3d MP2RAGE) and (sagittal T2w, sagittal STIR, 3D MP2RAGE). The performances of the methods were then assessed using a dedicated testing set. The training and testing data sets were manually annotated using a principled process involving 5 experts for each case. All pipelines were submitted in the form of a docker and integrated to the VIP platform <fnref n="4" /> and participants were not involved in the evaluation on the test set data. The dataset can be requested at https://shanoir.irisa.fr/shanoir-ng/welcome and we hope it will contribute to unlock new solutions to improve spinal cord lesion segmentation methods.

**Acknowledgments** : Data collection has been supported by a grant provided by the French State and handled by the "Agence Nationale de la Recherche," within the framework of the "France 2030" programme, under the reference ANR-10-COHO-002" OFSEP; and the support of the "Eugène Devic EDMUS Foundation against multiple sclerosis and by the OFSEP national project in France (ANR-10-COHO-002). In addition, the challenge was partially supported by the France Life Imaging national project in France (ANR-11-INBS-006) as well as through the France 2030 programme (ANR-21-RHUS-0014).

<footnote n="1">¹https://portal.fli-iam.irisa.fr/MS-Multi-Spine/</footnote>

<footnote n="2">²https://www.ofsep.org/en</footnote>

<footnote n="3">³https://portal.fli-iam.irisa.fr/</footnote>

<footnote n="4">⁴creatis.insa-lyon.fr/vip/</footnote>
# Organization

## Ms-multi-spine organization team

<table><tr><td>Romain Casey</td><td>OFSEP, Lyon, France</td></tr><tr><td>Benoit Combès</td><td>Inria Rennes, France</td></tr><tr><td>François Cotton</td><td>University Hospital of Lyon, France</td></tr><tr><td>Michel Dojat</td><td>Inserm GIN, Grenoble, France</td></tr><tr><td>Michael Kain</td><td>Inria Rennes, France</td></tr><tr><td>Anne Kerbrat</td><td>University Hospital of Rennes, France</td></tr><tr><td>Sorina Pop</td><td>Université Claude Bernard, Lyon, France</td></tr></table>

## Technical and scientific support

<table><tr><td>Gwenael Ambrosino</td><td>Université Claude Bernard, Lyon, France</td></tr><tr><td>Axel Bonnet</td><td>Université Claude Bernard, Lyon, France</td></tr><tr><td>Cédric Meurée</td><td>Inria Rennes, France</td></tr><tr><td>Louis Padé</td><td>Inria Rennes, France</td></tr><tr><td>Gwendal Soisnard</td><td>Inria Rennes, France</td></tr></table>
# Contents

<table><tr><td>SynSpine-MS: Enhancing Multi-Modal Spinal Multiple Sclerosis Detection via Syn- thetic MRI Generation<br>A. Aly, M. Salem, S. Hassan, and M. Yaqub</td><td>6</td></tr><tr><td>Multimodal pipeline for multiple sclerosis spinal cord lesion segmentation leveraging single-input model for instance segmentation with corresponding probability scores<br>P. Benveniste, T. Dagonneau, H. Lombaert, and J. Cohen-Adad</td><td>9</td></tr><tr><td>Team NeuroPoly: Description of the multimodal pipeline using a 2-channel T2w + extraSequence nnUNet for the The MS-Multi-Spine Miccai 2025 Challenge<br>T. Dagonneau, P. Benveniste, M. Guay-Paquet, and J. Cohen-Adad</td><td>17</td></tr><tr><td>Late fusion for multi-sequence MS lesion segmentation in spinal cord MRI<br>R. Walsh, F. Galassi, and B. Combès</td><td>23</td></tr><tr><td>Modular Cross-Attention Fusion for Multi-Modal MS Spine Lesion Segmentation with Missing Modalities<br>J. Wang, H. Liu, H. Li, F. Bagnato, and I. Oguz</td><td>28</td></tr></table>
# SynSpine-MS: Enhancing Multi-Modal Spinal
Multiple Sclerosis Detection via Synthetic MRI
Generation

Ahmed Aly¹, Mostafa Salem¹,², Salma Hassan¹, and Mohammad Yaqub¹

¹ Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

² Faculty of Computers and Information, Assiut University, Assiut, Egypt

**Abstract.** Missing MRI modalities can significantly compromise the detection of spinal cord lesions in Multiple Sclerosis (MS). We present **SynSpine-MS**, a two-stage pipeline that first synthesizes missing modalities and then proposes a segmentation method from real and synthetic scans. Our approach improves robustness to incomplete data and achieves accurate lesion-level predictions through joint synthesis and segmentation.

**Keywords:** Multiple Sclerosis · Modality Synthesis · Lesion Detection

## 1 Introduction

Spinal cord lesion segmentation is essential for MS diagnosis, prognosis, and monitoring [1]. MRI is the standard imaging modality, but missing sequences are common due to scan time, artifacts, or protocol limitations, reducing the performance of multi-modal models.

We propose **SynSpine-MS**, a two-stage pipeline that synthesizes missing modalities and segments lesions using a 3D U-Net. Developed for the MS-Multi-Spine Challenge [1], it improves robustness to incomplete inputs by integrating synthetic and acquired data.

## 2 Methodology

### MRI Seq2Seq Generation

MRISeq2Seq performs modality imputation by synthesizing missing MRI sequences from available ones. It uses a Commonality-and-Discrepancy-Sensitive Encoder to capture both shared and modality-specific features [3], and a Dynamic Feature Unification Module for soft and hard fusion across inputs. The decoder reconstructs all target modalities in a multi-output setting, enabling generalization to arbitrary input configurations. Training involves adversarial loss with multiple discriminators and random input masking to simulate missing-modality scenarios, enabling flexible and accurate modality translation. This
Seq2Seq-based design follows the framework introduced in [3], which has also been released as a public implementation. This imputation process helps recover critical imaging contrasts that may be unavailable due to acquisition constraints or artifacts. Reconstructing a complete modality set provides richer, more consistent inputs for lesion segmentation, enhancing accuracy and robustness on incomplete scans (Fig. 1a). Fig. 1b shows output examples from our generation model.

## 3D U-Net Segmentation

After synthesizing the missing modalities, we train a 3D U-Net model with four input channels (one per available or imputed modality) to segment spinal cord lesions using the combined original and generated modalities as input. The network has five resolution levels and is trained with a Dice + Cross-Entropy hybrid loss. Dropout regularization ($p = 0.2$) is applied in the bottleneck. Optimization is performed using SGD with Nesterov momentum (0.99) and a polynomial learning rate decay (initial learning rate 0.01). Training is run for up to 1,000 epochs, and the checkpoint with the best validation loss is retained as the final model per fold. We use a patch size of 256 × 64 × 128, batch size of 2, and apply z-score normalization per modality with resampling to 0.5 mm isotropic spacing. Five models are trained using 5-fold cross-validation with an 80/20 train/val split. At inference, outputs are ensembled via soft majority voting. A configuration and hyperparameter sweep is performed using the nnU-Net framework [2]. This corresponds to Stage 2 of our pipeline (Fig. 1a).

## Post-processing and Detection

At inference time, we apply test-time augmentation using mirroring and Gaussian importance weighting. The predicted softmax outputs are thresholded at 0.5 to generate binary lesion masks. Instance segmentation is performed via 3D connected component analysis with 26-connectivity. For each segmented lesion, we compute the mean softmax probability across its voxels as a confidence estimate.

## References

1. France Life Imaging – IAM, IRISA: MS-Multi-Spine Challenge. https://portal.fli-iam.irisa.fr/ms-multi-spine/ (accessed 2025-06-14)

2. Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. *Nature Methods*, **18**(2), 203–211 (2021)

3. Han, L., Tan, T., Zhang, T., Huang, Y., Wang, X., Gao, Y., Teuwen, J., Mann, R.: Synthesis-based imaging-differentiation representation learning for multi-sequence 3D/4D MRI. Medical Image Analysis **92**, 103044 (2024)
<figure><img src="image_4.png" /><figcaption>Fig. 1: (a) The two-stage SynSpine-MS pipeline. (b) Output examples from the synthesis stage.</figcaption></figure>
Multimodal pipeline for multiple sclerosis spinal
cord lesion segmentation leveraging single-input
model for instance segmentation with
corresponding probability scores

Pierre-Louis Benveniste¹, Thomas Dagonneau¹, Hervé Lombaert¹, and Julien Cohen-Adad¹

Polytechnique Montreal, Montreal, QC, Canada

**Abstract.** This paper presents the complete pipeline developed for the MS-Multi-Spine MICCAI 2025 challenge, addressing the task of automated segmentation of multiple sclerosis lesions in the spinal cord on multi-modal data. The proposed approach is based on a 3D Residual-Encoder U-Net trained on single-modality inputs. A comprehensive post-processing framework is applied to the model outputs, producing lesion-wise instance segmentations along with probability scores.

**Keywords:** Multiple sclerosis · Lesion segmentation · Multimodal · Instance segmentation.

# 1 Introduction

Multiple sclerosis (MS) is a chronic, progressive disease that primarily affects the central nervous system, particularly targeting the brain, spinal cord, and optic nerves. Magnetic resonance imaging (MRI) has become the cornerstone of MS diagnosis and monitoring, due to its high sensitivity in detecting lesions [24]. While brain imaging has historically dominated MS research, spinal cord imaging is now gaining increasing attention due to its strong clinical relevance. However, imaging the spinal cord remains challenging due to its small cross-sectional area, sensitivity to motion, and considerable variability in acquisition protocols across sites. As a result, a wide variety of MRI sequences are employed – often combining multiple contrasts and orientations (e.g., axial, sagittal, and 3D) – to optimise lesion visibility [9, 11, 12, 23]. Precise tracking of disease progression requires precise lesion segmentation tools able to segment lesions on multiple contrasts. While numerous automated tools have been developed for brain MS lesion segmentation [17, 2, 26], few exist for the spinal cord, and none currently support segmentation of spinal cord lesions across heterogeneous acquisitions for the same patient. Furthermore, high inter- and intra-rater variability in manual annotations poses an additional challenge [27], highlighting the importance of robust automated methods.

In this study, we introduce a novel pipeline for the automatic segmentation of MS lesions in the spinal cord across multiple MRI acquisitions from the same
patient. The pipeline integrates post-processing of model predictions to produce
lesion instance segmentations, along with associated probability scores, enabling
both robust lesion detection and interpretable uncertainty quantification.

## 2 Related works

Recent literature has seen substantial progress in the development of automated methods for segmenting MS lesions, with a majority of efforts concentrated on brain MRI. Numerous studies have demonstrated the effectiveness of deep learning approaches for lesion segmentation on single brain scans [17, 2, 26]. Beyond single-scan settings, several methods have leveraged multiple MRI acquisitions of the same subject to improve segmentation accuracy. For instance, in [8], they proposed a clustering-based approach for segmenting MS lesions on co-registered brain scans. Similarly, deep learning strategies have been applied to registered multimodal inputs, including convolutional neural networks trained on combinations of contrasts [29], diffusion models [22], and self-supervised frameworks using stacked registered modalities [13]. These methods, however, rely heavily on image registration, which is generally reliable in the brain due to its relatively consistent anatomy and large structures. In contrast, spinal cord MRI presents unique challenges for registration owing to its small diameter, high curvature, and pronounced susceptibility to motion and distortion artefacts. Misregistration in this context can significantly degrade the visibility of lesions and lead to loss of critical spatial information.

Segmentation of MS lesions in the spinal cord has received comparatively less attention. Existing methods typically operate on monomodal input and are tailored to specific MRI sequences. For example, segmentation tools have been proposed for T2-weighted and T2*-weighted scans [14], MP2RAGE [20], PSIR or STIR sequences [5] and T2w axial scans [21]. Recently, in [4], a multi-contrast single-input model was proposed. Notably, only one reported spinal cord method attempts to use dual-contrast input [19], but the model is not publicly available, limiting reproducibility and adoption.

To date, all state-of-the-art spinal cord segmentation methods are based on convolutional neural networks (CNNs). Empirical evidence suggests that well-optimised, classical architectures often outperform more recent, complex architectures, as demonstrated by the widespread adoption of the nnUNet framework [16]. Most methods report Dice similarity coefficient as the primary evaluation metric; while informative, this metric has known limitations in handling small lesion volumes and imbalanced data distributions, common characteristics in spinal cord MS lesion datasets. In this study, we adopted the nnU-Net framework for model development and complemented traditional voxel-wise metrics with lesion-wise evaluation, enabling a more clinically relevant assessment of lesion detection performance.
# 3 Materials and methods

## 3.1 Data

MRI data came from the MS-Multi-Spine Miccai 2025 challenge [1]. The training dataset comprised spinal cord MRI scans from 100 MS subjects [7]. Three contrast pairings were included: 50 T2w/STIR pairs, 25 T2w/PSIR pairs, and 25 T2w/MP2RAGE pairs. Images were acquired on 21 different scanners, both 1.5T (n=104) and 3T (n=96), from two manufacturers: Philips (n=58) and Siemens (n=142). T2w, PSIR and STIR sequences consisted of 2D sagittal acquisitions with 3mm thick slices, while MP2RAGE sequences consisted of 3D acquisition at 1.0 x 0.9 x 0.9 mm. Across all modalities, spatial resolution was 2.78±0.81 × 0.59±0.21 × 0.59±0.21 mm, reported in RPI orientation.

## 3.2 Preprocessing

The MS-Multi-Spine challenge organisers provided a preprocessed version of the data (including notably zeroing voxels outside the spine region), which we used for training our model. T2w raw images served as the reference space for multimodal alignment, consistent with the annotation space in which lesion segmentations were originally performed. The preprocessed and registered lesion segmentations were not utilised, as visual inspection revealed poor alignment between lesion segmentations and the corresponding registered contrasts, likely due to the absence of appropriate resampling during the preprocessing pipeline. Registration was performed using the Spinal Cord Toolbox (SCT) [10]. For T2w scans, registration to the raw space was performed using the Qform matrix (via the command "sct_register_multimodal -identity 1"). For all other contrasts, registration to the corresponding T2w raw space was conducted using a contrast-agnostic, deep learning-based algorithm [3] built on the SynthMorph architecture [15] (via "sct_register_multimodal -param step=1,type=im,algo=dl"). The quality of the registration was visually assessed.

## 3.3 Data augmentations

Given the pronounced class imbalance inherent in spinal cord lesion segmentation, where lesion voxels represent only a minute fraction of the total volume, as well as the limited size of the dataset (100 subjects), an aggressive data augmentation strategy was employed to improve model generalizability and prevent overfitting. The augmentation protocol followed the approach adopted in the TotalSpineSeg work [28]. Data augmentation included a set of image-level intensity augmentations, including Scharr filtering, non-linear function transformations (e.g., square root, sine, exponential, sigmoid), histogram equalization, and contrast manipulation. Several augmentations were specifically designed to simulate variability in scanner acquisition and pathology presentation, such as motion artifacts, ghosting, spikes, bias field inhomogeneities, Gaussian blurring, and additive noise. Brightness and contrast adjustments were introduced through multiplicative brightness scaling, gamma correction, and contrast transformation.
Spatial diversity was further enhanced through affine and elastic transformations. Additionally, augmentations using the mask were applied, including label redistribution and synthetic image generation from binary masks.

## 3.4 Model and architecture

The model uses a Residual-Encoder-UNet architecture. This architecture is a fully convolutional encoder-decoder network composed of six resolution stages. Each encoder stage consists of residual blocks with increasing numbers of feature channels: 32, 64, 128, 256, 320, and 320, respectively. Downsampling is achieved via strided convolutions, and kernel sizes are fixed at 3 × 3 × 3 throughout. Instance normalisation (InstanceNorm3d) and Leaky ReLU activations are used throughout the network. The model processes 3D input patches of size 192 × 256 × 160 voxels. Given the anisotropic nature of spinal cord images, spacing was standardised to 0.677 × 0.5 × 0.5625 mm. The architecture was implemented using PyTorch and integrated into the nnUNetv2 framework with the nnUNetPlannerResEncL planner.

## 3.5 Training and evaluation

Model training was conducted using a five-fold cross-validation strategy. The data were initially split at the subject level into 80% training and 20% validation sets, while maintaining the same distribution of contrast pairings across splits. Model training was performed using the Dice Cross-Entropy loss (DiceCELoss) without smoothing. Similarly to [25], batch sampling was weighted inversely to the square root of the number of samples per contrast group. A batch size was set to 2 to maximise model generalizability and prevent overfitting [18]. Training was conducted for 500 epochs, without any pretraining. Data augmentation was applied on-the-fly during training. Normalisation was performed channel-wise using Z-score normalisation. An initial learning rate of 1e-2 was used with a weight decay of 3e-5. Optimization was performed using stochastic gradient descent (SGD) with Nesterov momentum (momentum = 0.99), and learning rate scheduling followed a polynomial decay policy implemented via PolyLRScheduler. During training, model performance was monitored using DiceCELoss and the best model was selected by Dice score evaluation on the validation set. For final performance assessment, both voxel-wise and lesion-wise metrics were employed. Voxel-wise overlap was quantified using the Dice similarity coefficient. To provide a clinically meaningful evaluation of lesion detection, lesion-wise metrics were computed based on a minimum intersection-over-union (IoU) threshold of 10%. These included lesion-wise positive predictive value (lesion-PPV), lesion-wise sensitivity, and lesion-wise F1 score, capturing the model's ability to correctly detect and localise individual lesions.

## 3.6 Post-processing

To enhance the precision of the segmentation outputs, a comprehensive post-processing pipeline was applied to the model predictions. This pipeline operated
on the soft segmentations obtained by averaging the outputs of the five cross-validation folds, applied independently to each input modality. (i) All predicted lesion connected components (connectivity = 6) located outside the spinal cord were suppressed by applying a spinal cord mask, dilated by 2 mm in the axial plane, obtained from [6]. (ii) Spurious low-confidence predictions were further eliminated by discarding lesion candidates whose maximum voxel value did not exceed a predefined threshold of 0.8. This threshold was empirically selected through a grid search over the range [0.5, 0.95] with a step size of 0.05. (iii) Modality predictions were fused via voxel-wise averaging of the respective soft segmentations. Given the differing fields of view across input modalities, a dilated spinal cord mask was used to estimate the anatomical coverage of each modality. Voxel averaging was weighted by the estimated coverage of the spinal cord in each modality. (iv) The resulting fused soft segmentation was binarised using a threshold of 0.5, empirically determined by evaluating performance metrics across thresholds ranging from 0.3 to 0.95 in steps of 0.05. (v) Small lesion candidates were excluded by applying a minimum volume criterion: lesions with volumes below 50 voxels were removed. This volume threshold was optimized by scanning the range [5, 60] voxels with a step size of 5 and selecting the value maximising lesion-wise F1 score. (vi) Instance segmentations were computed by assigning a unique label to each lesion. A lesion-level confidence score was computed for each instance as the mean voxel intensity within the soft segmentation.

# 4 Discussion

In this study, we introduced a multimodal pipeline for spinal cord multiple sclerosis lesion segmentation, combining a single-modality 3D model with a structured post-processing framework to produce instance-level predictions enriched with lesion-wise probability scores.

Surprisingly, fine-tuning the pretrained spinal cord MS lesion segmentation model from [4] did not improve performance. This outcome may reflect domain shift between datasets, overfitting during adaptation, or suboptimal transfer of pretrained features. These results highlight the limitations of direct fine-tuning in the presence of acquisition heterogeneity.

Our results demonstrate that high-quality lesion segmentation can be achieved without relying on multimodal input at inference time. The integration of modality fusion and calibrated post-processing steps contributed to both the robustness and interpretability of predictions. High lesion-wise performance metrics observed on the training set confirm the pipeline's capacity to detect and localise individual lesions with high confidence.

# References

1. The MS-multi-spine miccai 2025 challenge. https://portal.fli-iam.irisa.fr/ms-multi-spine/, accessed: 2025-6-12
2. Aslani, S., Dayan, M., Storelli, L., Filippi, M., Murino, V., Rocca, M.A., Sona, D.: Multi-branch convolutional neural network for multiple sclerosis lesion segmentation. Neuroimage **196**, 1–15 (Aug 2019)

3. Beal, E., NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montreal, Cohen-Adad, J.: Contrast-agnostic deep learning-based registration pipeline: Validation in spinal cord multimodal MRI data. Aperture Neuro **3**(2023), 1–21 (2023)

4. Benveniste, P.L., Lee, L.E., Prat, A., Vavasour, Z., Tam, R., Traboulsee, A., Kolind, S., Oh, J., Chen, M., Tsagkas, C., Granziera, C., Medina, N.L., Muhlau, M., Kirschke, J., Mc Ginnis, J., Reich, D.S., Hemond, C., Callot, V., Demortière, S., Audoin, B., Nair, G., Filippi, M., Valsasina, P., Rocca, M.A., Ciccarelli, O., Yiannakas, M., Kearney, H., Granberg, T., Ouellette, R., Stawiarz, L., Hillert, J., Shahamat, T., Bakshi, R., Mainero, C., Treaba, C.A., Kerbrat, A., Bannier, E., Edan, G., Labauge, P., O'Grady, K., Smith, S.A., Shepherd, T.M., Charlson, E., Brisset, J.C., Talbott, J., Liu, Y., Lombaert, H., Cohen-Adad, J.: Automatic multi-contrast MRI segmentation of spinal cord lesions. In: 10th annual Americas Committee for Treatment and Research in Multiple Sclerosis (ACTRIMS) Forum 2025, West Palm Beach, Florida. (2025)

5. Benveniste, P.L., Valošek, J., Chen, M., Molinier, N., Lee, L.E., Prat, A., Vavasour, Z., Tam, R., Traboulsee, A., Kolind, S., Oh, J., Cohen-Adad, J.: Automatic segmentation of spinal cord multiple sclerosis lesions across multiple sites, contrasts and vendors. In: ISMRM Annual Meeting. ISMRM, Concord, CA (Nov 2024)

6. Bédard, S., Karthik, E.N., Tsagkas, C., Pravatà, E., Granziera, C., Smith, A., Weber, Ii, K.A., Cohen-Adad, J.: Towards contrast-agnostic soft segmentation of the spinal cord. Med. Image Anal. **101**(103473), 103473 (Apr 2025)

7. Casey, R., Combès, B., Cotton, F., Dojat, M., Kain, M., Kerbrat, A., Pop, S.: Multiple sclerosis spinal cord lesions detection from MultiSequence MRIs challenge (MS-multi-spine) (2024)

8. Cetin, O., Seymen, V., Sakoglu, U.: Multiple sclerosis lesion detection in multimodal MRI using simple clustering-based segmentation and classification. Inform. Med. Unlocked **20**(100409), 100409 (2020)

9. Cohen-Adad, J., Alonso-Ortiz, E., Abramovic, M., Arneitz, C., Atcheson, N., Barlow, L., Barry, R.L., Barth, M., Battiston, M., Büchel, C., Budde, M., Callot, V., Combes, A.J.E., De Leener, B., Descoteaux, M., de Sousa, P.L., Dostál, M., Doyon, J., Dvorak, A., Eippert, F., Epperson, K.R., Epperson, K.S., Freund, P., Finsterbusch, J., Foias, A., Fratini, M., Fukunaga, I., Wheeler-Kingshott, C.A.M.G., Germani, G., Gilbert, G., Giove, F., Gros, C., Grussu, F., Hagiwara, A., Henry, P.G., Horák, T., Hori, M., Joers, J., Kamiya, K., Karbasforoushan, H., Keřkovský, M., Khatibi, A., Kim, J.W., Kinany, N., Kitzler, H., Kolind, S., Kong, Y., Kudlička, P., Kuntke, P., Kurniawan, N.D., Kusmia, S., Labounek, R., Laganà, M.M., Laule, C., Law, C.S., Lenglet, C., Leutritz, T., Liu, Y., Llufriu, S., Mackey, S., Martinez-Heras, E., Mattera, L., Nestrasil, I., O'Grady, K.P., Papinutto, N., Papp, D., Pareto, D., Parrish, T.B., Pichiecchio, A., Prados, F., Rovira, Ruitenberg, M.J., Samson, R.S., Savini, G., Seif, M., Seifert, A.C., Smith, A.K., Smith, S.A., Smith, Z.A., Solana, E., Suzuki, Y., Tackley, G., Tinnermann, A., Valošek, J., Van De Ville, D., Yiannakas, M.C., Weber, 2nd, K.A., Weiskopf, N., Wise, R.G., Wyss, P.O., Xu, J.: Generic acquisition protocol for quantitative MRI of the spinal cord. Nat. Protoc. **16**(10), 4611–4632 (Oct 2021)
10. De Leener, B., Lévy, S., Dupont, S.M., Fonov, V.S., Stikov, N., Louis Collins, D., Callot, V., Cohen-Adad, J.: SCT: Spinal cord toolbox, an open-source software for processing spinal cord MRI data. Neuroimage (Jan 2017)

11. Demortière, S., Lehmann, P., Pelletier, J., Audoin, B., Callot, V.: Improved cervical cord lesion detection with 3D-MP2RAGE sequence in patients with multiple sclerosis. AJNR Am. J. Neuroradiol. 41(6), 1131–1134 (Jun 2020)

12. Fechner, A., Savatovsky, J., Saragoussi, E., Roux, P., Sadik, J.C., Lecler, A.: Intérêt du 3D PSIR pour la détection de lésions médullaires de sclérose en plaque. J. Neuroradiol. 44(2), 77–78 (Mar 2017)

13. Gjergji, A., Ciranni, M., Moro, M., Murino, V., Pastore, V.P.: Self-supervised multi-modal learning for accurate MRI multiple sclerosis segmentation. In: 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI). pp. 1–4. IEEE (Apr 2025)

14. Gros, C., De Leener, B., Badji, A., Maranzano, J., Eden, D., Dupont, S.M., Talbott, J., Zhuoquiong, R., Liu, Y., Granberg, T., Ouellette, R., Tachibana, Y., Hori, M., Kamiya, K., Chougar, L., Stawiarz, L., Hillert, J., Bannier, E., Kerbrat, A., Edan, G., Labauge, P., Callot, V., Pelletier, J., Audoin, B., Rasoanandrianina, H., Brisset, J.C., Valsasina, P., Rocca, M.A., Filippi, M., Bakshi, R., Tauhid, S., Prados, F., Yiannakas, M., Kearney, H., Ciccarelli, O., Smith, S., Treaba, C.A., Mainero, C., Lefeuvre, J., Reich, D.S., Nair, G., Auclair, V., McLaren, D.G., Martin, A.R., Fehlings, M.G., Vahdat, S., Khatibi, A., Doyon, J., Shepherd, T., Charlson, E., Narayanan, S., Cohen-Adad, J.: Automatic segmentation of the spinal cord and intramedullary multiple sclerosis lesions with convolutional neural networks. Neuroimage 184, 901–915 (Jan 2019)

15. Hoffmann, M., Billot, B., Greve, D.N., Iglesias, J.E., Fischl, B., Dalca, A.V.: SynthMorph: Learning contrast-invariant registration without acquired images. IEEE Trans. Med. Imaging 41(3), 543–558 (Mar 2022)

16. Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU-net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18(2), 203–211 (Feb 2021)

17. Kaur, A., Kaur, L., Singh, A.: State-of-the-art segmentation techniques and future directions for multiple sclerosis brain lesions. Arch. Comput. Methods Eng. 28(3), 951–977 (May 2021)

18. Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P.T.P.: On large-batch training for deep learning: Generalization gap and sharp minima. arXiv [cs.LG] (Sep 2016)

19. Lodé, B., Hussein, B.R., Meurée, C., Walsh, R., Gaubert, M., Lassalle, N., Courbon, G., Martin, A., Le Bars, J., Durand-Dubief, F., Bourre, B., Maarouf, A., Outeryck, O., Mehier, C., Poulin, A., Cathelineau, C., Hong, J., Criton, G., Motillon-Alonso, S., Lecler, A., Charbonneau, F., Duron, L., Bani-Sadr, A., Delpierre, C., Ferré, J.C., Edan, G., Cotton, F., Casey, R., Galassi, F., Combès, B., Kerbrat, A.: Evaluation of a deep learning segmentation tool to help detect spinal cord lesions from combined T2 and STIR acquisitions in people with multiple sclerosis. Eur. Radiol. (Apr 2025)

20. Medina, N.L.: model_seg_mspekrage: Model repository for MS lesion segmentation on MP2RAGE data from university of Basel

21. Naga Karthik, E., McGinnis, J., Wurm, R., Ruehling, S., Graf, R., Valosek, J., Benveniste, P.L., Lauerer, M., Talbott, J., Bakshi, R., Tauhid, S., Shepherd, T., Berthele, A., Zimmer, C., Hemmer, B., Rueckert, D., Wiestler, B., Kirschke, J.S., Cohen-Adad, J., Mühlau, M.: Automatic segmentation of spinal
cord lesions in ms: A robust tool for axial t2-weighted mri scans. Imaging Neuroscience **3**, IMAG.a.45 (06 2025). https://doi.org/10.1162/IMAG.a.45, https://doi.org/10.1162/IMAG.a.45

22. Rondinella, A., Guarnera, F., Giudice, O., Ortis, A., Russo, G., Crispino, E., Papalardo, F., Battiato, S.: Enhancing multiple sclerosis lesion segmentation in multimodal MRI scans with diffusion models. In: 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). pp. 3733–3740. IEEE (Dec 2023)

23. Saslow, L., Li, D.K.B., Halper, J., Banwell, B., Barkhof, F., Barlow, L., Costello, K., Damiri, P., Dunn, J., Giri, S., Maes, M., Morrow, S.A., Newsome, S.D., Oh, J., Paul, F., Quarterman, P., Reich, D.S., Shewchuk, J.R., Shinohara, R.T., Van Hecke, W., van de Ven, K., Wallin, M.T., Wolinsky, J.S., Traboulsee, A.: An international standardized magnetic resonance imaging protocol for diagnosis and follow-up of patients with multiple sclerosis: Advocacy, dissemination, and implementation strategies. Int. J. MS Care **22**(5), 226–232 (Sep 2020)

24. Thompson, A.J., Banwell, B.L., Barkhof, F., Carroll, W.M., Coetzee, T., Comi, G., Correale, J., Fazekas, F., Filippi, M., Freedman, M.S., Fujihara, K., Galetta, S.L., Hartung, H.P., Kappos, L., Lublin, F.D., Marrie, R.A., Miller, A.E., Miller, D.H., Montalban, X., Mowry, E.M., Sorensen, P.S., Tintoré, M., Traboulsee, A.L., Trojano, M., Uitdehaag, B.M.J., Vukusic, S., Waubant, E., Weinshenker, B.G., Reingold, S.C., Cohen, J.A.: Diagnosis of multiple sclerosis: 2017 revisions of the McDonald criteria. Lancet Neurol. **17**(2), 162–173 (Feb 2018)

25. Ulrich, C., Isensee, F., Wald, T., Zenk, M., Baumgartner, M., Maier-Hein, K.H.: MultiTalent: A multi-dataset approach to medical image segmentation. In: Lecture Notes in Computer Science, pp. 648–658. Lecture notes in computer science, Springer Nature Switzerland, Cham (2023)

26. Valverde, S., Cabezas, M., Roura, E., González-Villà, S., Pareto, D., Vilanova, J.C., Ramió-Torrentà, L., Rovira, J., Oliver, A., Lladó, X.: Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach. Neuroimage **155**, 159–168 (Jul 2017)

27. Walsh, R., Meurée, C., Kerbrat, A., Masson, A., Hussein, B.R., Gaubert, M., Galassi, F., Combés, B.: Expert variability and deep learning performance in spinal cord lesion segmentation for multiple sclerosis patients. In: 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS). pp. 463–470. IEEE (Jun 2023)

28. Warszawer, Y., Molinier, N., Valošek, J., Bédard, S., Shirbint, E., Benveniste, P.L., Achiron, A., Eshaghi, A., Cohen-Adad, J.: TotalSpineSeg: Robust segmentation and labeling of vertebrae, intervertebral discs, spinal cord, and spinal canal in MRI images using nnU-net and iterative algorithm (2024)

29. Wynen, M., Gordaliza, P.M., Stölting, A., Maggi, P., Bach Cuadra, M., Macq, B.: Multi-modal segmentation for paramagnetic rim lesion detection in multiple sclerosis. In: Wu, S., Yoshida, H. (eds.) Medical Imaging 2024: Imaging Informatics for Healthcare, Research, and Applications. SPIE (Apr 2024)
# Team NeuroPoly: Description of the multimodal pipeline using a 2-channel T2w+extraSequence nnUNet for the The MS-Multi-Spine Miccai 2025 Challenge

Thomas Dagonneau¹,²,³, Pierre-Louis Benveniste¹,², Mathieu Guay-Paquet¹, and Julien Cohen-Adad¹,²,⁴,⁵

¹ NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montreal,
Montreal, QC, Canada

² Mila - Quebec AI Institute, Montreal, QC, Canada

³ École Polytechnique, Palaiseau, France

⁴ Functional Neuroimaging Unit, CRIUGM, Université de Montréal, Montreal, QC,
Canada

⁵ Centre de Recherche du CHU Sainte-Justine, Université de Montréal, Montreal,
QC, Canada

**Abstract.** This paper presents the complete pipeline developed by Neu-
ropoly for the MS-Multi-Spine MICCAI 2025 challenge, aimed at the au-
tomated segmentation of multiple sclerosis lesions in the spinal cord using
multimodal MRI data. The approach uses a 2D Residual-Encoder nnU-
Net trained on multi-contrast inputs with 2-channel T2w+extraSequence.
A detailed post-processing pipeline is applied to the model predictions,
generating lesion-wise instance segmentations along with corresponding
probability scores. The code is available here: https://github.com/ivadomed/ms-
multi-spine-challenge-2024

**Keywords:** ms lesions · segmentation · multimodal

# 1 Introduction

Multiple sclerosis (MS) is a chronic disease of the central nervous system, with spinal cord involvement playing a critical role in clinical outcomes. Although MRI is essential for MS diagnosis and disease monitoring [1], spinal cord imaging remains challenging due to anatomical complexity and variability in acquisition protocols. Existing automated lesion segmentation tools have focused primarily on the brain [2–4], and very few methods were designed for spinal cord lesions, particularly across heterogeneous and multimodal MRI data. Additionally, high inter- and intra-rater variability in manual annotations [5] highlights the need for reliable, automated segmentation pipelines. In this study, we introduce a novel pipeline for the automatic segmentation of MS lesions in the spinal cord using multiple MRI acquisitions from the same patient. Our approach combines
a 2-channel T2w+extraSequence nnUNet with a robust post-processing framework to produce accurate lesion instance segmentations and estimate prediction uncertainty, addressing the specific challenges of spinal cord imaging.

# 2 Method

## 2.1 Data

The total dataset consists of 200 patients (100 for training and 100 for testing), each with one T2w image and one additional image from among PSIR, STIR, or MP2RAGE. In the training set, there are 50 patients with the T2w+STIR combination, 25 with T2w+PSIR, and 25 with T2w+MP2RAGE. The testing set (hidden from us) includes 40 T2w+STIR, 20 T2w+PSIR, 20 T2w+MP2RAGE, and 20 T2w+STIR+MP2RAGE combinations. Each patient has one ground truth segmentation annotated by the challenge organizers on the T2w image.

Data were acquired on Philips (n=58) and Siemens (n=142) scanners, at 1.5T (n=104) and 3T (n=96) field strengths.

The average voxel spacing in RPI (mean ± std, in mm) per modality is as follows:

- **All modalities (combined):** 2.78 ± 0.81; 0.59 ± 0.21; 0.59 ± 0.21

- **T2w:** 3.02 ± 0.32; 0.51 ± 0.13; 0.51 ± 0.13

- **PSIR:** 2.74 ± 0.89; 0.50 ± 0.21; 0.50 ± 0.21

- **STIR:** 3.22 ± 0.34; 0.61 ± 0.19; 0.61 ± 0.19

- **MP2RAGE:** 1.0 ± 0.0; 0.94 ± 0.0; 0.94 ± 0.0

Each image and corresponding segmentation are available in three formats:

- **Raw:** unprocessed original scans

- **Preproc:** images cropped around the spinal cord with fixed margins and resampling to 0.5 mm isotropic

- **PreprocReg:** the second modality registered to the T2w image within the preproc space

We used the preproc images to reduce surrounding noise.

## 2.2 Preprocessing

All preprocessing steps were performed using the Spinal Cord Toolbox (SCT, 7.0) [6], working with the preproc images to maintain alignment with the T2w raw space while removing irrelevant surrounding tissue.

The preprocessing pipeline involved the following steps:

- **Spinal Cord Segmentation:** We first segmented the spinal cord on the T2w image using SCT's *deepseg spinalcord* model [7]. This segmentation was used both for later lesion filtering.

- **Image Registration:** The second modality (STIR, PSIR, or MP2RAGE) was registered to the T2w image using `sct_register_multimodal`.
- **Image Cropping:** We only considered voxels around the spinal cord region. We identified the minimum and maximum non-zero voxel indices across the volume and applied cropping using `sct_image`.

- **Data Formatting:** After preprocessing, all images and masks file names were formatted to comply with nnU-Net's input structure.

## 2.3 Model Training and Inference Pipeline

We trained a 2D multimodal nnU-Net v2 model [8] using the DA5 trainer (see details below) over 150 epochs with Dice loss (no smoothing). Most hyperparameters were optimized using nnU-Net's built-in heuristics. However, we manually defined the data splits to ensure balanced representation of image contrasts across folds, as nnU-Net's default splitting strategy is not contrast-aware. The resulting hyperparameters are the following:

- **UNet number of levels:** 7

- **Batch size:** 64

- **Patch size:** 640x224

- **Optimization:** SGD optimizer with an initial learning rate of 0.01 with a polynomial learning rate scheduler and a momentum of 0.99

- **Dropout:** no dropout

Initially, we trained models on four folds using 80 patients (60 for training, 20 for validation), while keeping an independent internal test set of 20 patients for evaluation and training strategy optimization (10 T2w+STIR, 5 T2w+PSIR, 5 T2w+MP2RAGE). This stage was used to fine-tune parameters related to preprocessing, data augmentation, and loss function.

The DA5 trainer applied standard 2D data augmentations during training, including random rotations, scaling, elastic deformations, flipping, Gaussian noise, brightness and contrast adjustments, and gamma correction—applied independently per modality to improve generalization.

After finalizing the training strategy, we performed a full five-fold training for submission, with each fold trained on 80 patients and validated on 20. The training parameters were chosen based on the test set in the previous step. For each fold we kept the model with the best validation Dice.

For inference, predictions were generated using all five models and combined using a pixelwise voting strategy to produce a single binary segmentation mask for each patient:

- **T2w+STIR:** A pixel was labelled positive if at least one model predicted it as such (i.e. softmax value>0.5), to address under-segmentation specifically observed for this modality pair.

- **T2w+MP2RAGE or T2w+PSIR:** A pixel was labelled positive only if at least three models agreed, to reduce over-segmentation specifically observed for these modality pairs.
Additionally, a soft segmentation was generated by averaging the five fold predictions. This soft probability map was subsequently used to estimate the probability of lesion presence.

In the context of the competition, when three images were provided, we performed inference only with two (T2w+MP2RAGE) because preliminary results suggest that the model performs better on this combination.

## 2.4 Post-processing

To generate submission-ready outputs, we performed several post-processing steps:

1. **Registration:** The predicted masks were registered to match the T2w raw space using `sct_register_multimodal`.

2. **Lesion Separation:** Connected components were computed to separate individual lesions with a connectivity of 6.

3. **Cleaning and Filtering:**

(a) Lesions outside the spinal cord were removed using a binary spinal cord mask. To avoid excluding true spinal lesions near the edge, the mask was dilated by 2 mm in all directions.

(b) Very small lesions (including isolated voxels) were removed using a minimum lesion size threshold of 18 voxels, determined empirically on the validation set.

(c) Lesions with a maximum voxel-wise probability below 0.4 in the soft segmentation map were excluded, as this indicates support from fewer than two models in the ensemble.

4. **Lesion Probability Estimation:** For each retained lesion, we calculated its probability score as the mean probability of its constituent pixels on the soft mask.

The final output of our pipeline consisted of a segmentation mask in `.nii.gz` format and a corresponding `.csv` file listing per-lesion probabilities, as required by the competition.

# 3 Discussion

Our proposed pipeline achieved the highest performance on our internal test set of 20 held-out subjects, outperforming the alternative strategies we also investigated in the context of this challenge. Some relevant hyperparameters and their impact on the performance are listed below:

- **Loss Function:** Using the standard Dice cross entropy loss instead of the Dice cross entropy loss without smoothing led to decreased segmentation quality.

- **Trainer Variants:** Alternative nnU-Net trainers, such as `nnUNetTrainer`, `nnUNetTrainerDA5ord0`, and `nnUNetTrainerDAExt`, did not outperform the DA5 trainer.
- **3D Model:** Switching to a 3D architecture yielded inferior results, likely due to the fact that ground truth lesions were annotated on the 2D sagittal T2w scan, which is highly anisotropic. Hence, when looking at the ground truth segmentations in the 1mm isotropic MP2RAGE, multiple adjacent slices exhibit the same segmentation shape, which is unrealistic and does not properly follow the contour of the lesions.

- **Channel Swapping:** Augmenting data by randomly switching the two input channels (T2w and second modality) led to degraded performance, possibly because the first channel was not contrast specific anymore.

- **Contrast-Specific Models:** Our proposed approach was to train a single multimodal model: T2w+? ("?" being MP2RAGE, STIR or PSIR). Training multiple multimodal models, i.e., one per second contrast (T2w+MPRAGE, T2w+STIR, T2w+PSIR), limited data availability for each model and resulted in poorer performance.

- **nnU-Net Fusion:** We also tested nnU-Net's built-in fusion strategy for combining predictions across folds. However, it is optimized using the validation set only and under-performed on our held-out test set.

- **Early Stopping:** Training for fewer than 30 epochs led to underfitting, while longer training allowed the model to converge effectively without significant overfitting.

- **Full Image Training:** Using the full image instead of cropped regions around the spine added irrelevant context and noise, which hurt performance.

An approach that showed promise but was ultimately not pursued for the final submission was a custom voting strategy. We observed that certain folds consistently performed better than others, and a highly tailored voting scheme improved performance on our internal test set. However, this method presented two main drawbacks:

1. **Reduced Training Data:** Custom voting requires maintaining separate fold-specific models, preventing the use of all available data for final training. This likely limits overall model capacity and robustness.

2. **Overfitting Risk:** Our internal test set, while useful for optimization, was relatively small. A highly optimized voting scheme might have overfit to this specific subset, reducing generalizability.

Considering these limitations, we opted for a more conservative and robust strategy: training on the full dataset using five folds and applying a simple, contrast-aware pixelwise voting scheme. This decision balanced performance, generalization, and data efficiency, aligning with the competition's constraints and objectives.

## References

1. Thompson, A.J., Banwell, B.L., Barkhof, F., Carroll, W.M., Coetzee, T., Comi, G., et al.: Diagnosis of multiple sclerosis: 2017 revisions of the McDonald criteria. Lancet Neurol. **17**, 162–173 (2018)
2. Valverde, S., Cabezas, M., Roura, E., González-Villà, S., Pareto, D., Vilanova, J.C., et al.: Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach. Neuroimage **155**, 159–168 (2017)

3. Aslani, S., Dayan, M., Storelli, L., Filippi, M., Murino, V., Rocca, M.A., et al.: Multi-branch convolutional neural network for multiple sclerosis lesion segmentation. Neuroimage **196**, 1–15 (2019)

4. Kaur, A., Kaur, L., Singh, A.: State-of-the-art segmentation techniques and future directions for multiple sclerosis brain lesions. Arch. Comput. Methods Eng. **28**, 951–977 (2021)

5. Walsh, R., Meurée, C., Kerbrat, A., Masson, A., Hussein, B.R., Gaubert, M., et al.: Expert variability and deep learning performance in spinal cord lesion segmentation for multiple sclerosis patients. In: 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS), pp. 369–374. IEEE (2023). https://doi.org/10.1109/cbms58004.2023.00263

6. De Leener, B., Lévy, S., Dupont, S.M., Fonov, V.S., Stikov, N., Collins, D.L., et al.: SCT: Spinal Cord Toolbox, an open-source software for processing spinal cord MRI data. Neuroimage **145**(Part A), 24–43 (2017)

7. Bédard, S., Karthik, E.N., Tsagkas, C., Pravatà, E., Granziera, C., Smith, A., et al.: Towards contrast-agnostic soft segmentation of the spinal cord. Med. Image Anal. **101**, 103473 (2025)

8. Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods **18**, 203–211 (2021)
Late fusion for multi-sequence MS lesion
segmentation in spinal cord MRI

Ricky Walsh¹, Francesca Galassi¹,†, and Benoît Combès¹,†

Univ Rennes, Inria, CNRS, Inserm, IRISA UMR 6074, Empenn, Rennes, France
{ricky.walsh,benoit.combes,francesca.galassi}@inria.fr

† Equal contribution

# 1 Introduction

Identifying lesions in spinal cord MRI is an important task for the diagnosis and prognosis of multiple sclerosis. However, this is a complex task and is subject to high inter-rater variability [6, 3]. Current guidelines recommend using several acquisitions with different contrasts for a given subject, but the heterogeneity of the possible sequences gives rise to a missing modality problem, where a particular sequence could be missing for a given subject, at training or at inference. A previous work [5] applied an intermediate fusion approach to tackle this missing modality problem and this showed limited success at inference, and therefore we opted here for a late fusion approach.

Our submission used a publicly available trained segmentation model [1] to obtain initial predictions. We then summarised the predicted probabilities and intensities in both available sequences for each predicted lesion, and we trained a classification model to predict whether a given instance is a true positive or false positive. In this way, we benefitted from the diversity of the large training set used to train the publicly available model, while also adapting the predictions to the characteristics of the challenge dataset and taking into account the particularities of each sequence.

# 2 Method

## 2.1 Deep Learning Segmentation Model

We used a publicly available trained model from the Spinal Cord Toolbox (SCT-v7.0) [2], specifically the contrast-agnostic model (sct_deepseg lesion_ms) [1]. This model was trained with the nnU-Net framework, and the dataset used for development was large and diverse, containing 2988 volumes, including 1806 T2-w, 286 PSIR, 265 MP2RAGE, and 72 STIR acquisitions, among other sequences. Preprocessing (e.g., reorientation and resampling) is handled by SCT in the background, as part of the nnU-Net framework. We extracted the softmax scores for all 100 cases, applying the model separately to the T2-w volume and the other sequence volume for each case; we will refer to the 3D image containing these softmax scores for each voxel as a 'pmap'. We also examined the impact
of test-time augmentation on the performance, i.e., flipping the input image on
each axis before obtaining a prediction, leading to eight possible versions of the
input, and thus eight pmap predictions, which were then averaged.

Using the preprocessed volumes as input to the SCT lesion segmentation
model led to a significantly reduced performance (a relative drop in FROC of
50%), presumably because the cropping applied before registration caused an
out-of-distribution effect. Therefore, we applied the lesion segmentation model
to the raw images, then computed the registration between the supplementary
acquisition to the T2-w volume, and used the resulting deformation field to map
the prediction into the T2-w space. We used the deep learning registration model
from SCT to compute this registration.

We masked predictions by the spinal cord segmentation to remove false pos-
itives, using the sagittal T2-w mask obtained from the SCT contrast-agnostic
spinal cord segmentation model. We further post-processed this mask with 3D
morphological closing using a cube element of length 3, took only the largest
3D connected component in the mask, and applied 2D opening to each sagittal
slice with a square element of length 3. To reduce the probability of some of the
spinal cord being left out of the mask, we took the union of the spinal cord mask
across all sagittal slices, and applied in-plane dilation with a square structuring
element of length 5.

## 2.2 Creating Lesion Instances

Given the pmaps obtained from the T2-w volume, the supplementary acquisition, and also the mean of these two pmaps, we binarised the scores at different thresholds to create segmentation masks. The tested thresholds were 10⁻⁶, 10⁻⁵, 10⁻⁴, 10⁻³, 10⁻², 0.1, and 0.5. Instances were then defined as 3D connected components (26-connectivity) in these binarised masks.

## 2.3 Feature Engineering

Next, we computed features at the instance-level, i.e., for each predicted lesion.
The features were calculated for each unique instance in each possible mask, i.e.,
a unique row in the resulting table of features is defined by the instance ID, the
mask type (T2-w, other sequence, or from the mean pmap), and the threshold
used to binarise the pmap. The computed features were as follows:

- **Predicted scores:** min, max, mean, median and the 10th, 20th, 90th, 95th, and 99th percentiles of the predicted softmax scores from the T2-w, other sequence, and mean pmap.

- **Intensity features:**
  - Min, max, mean, median, 5th and 95th percentiles of the intensities in the T2-w and other sequence acquisitions, after normalising using the mean and standard deviation of the spinal cord intensities.
- We also computed the mean contrast of the lesion vs. the spinal cord, the ratio of mean lesion intensity over spinal cord intensity standard deviation, and the mean contrast with respect to its neighbourhood, defined as a 3D dilation with a ball of radius 5 voxels.

- **Lesion volume and load:** total predicted load and the volume of the current lesion in the T2-w, other sequence, and mean masks.

- **Distance features:** distance to middle slice of spinal cord (in mm) and distance to top of spinal cord (in mm).

- **IoU features:** intersection over union with respect to the prediction in the other sequence pmap and mean pmap.

- **Other sequence:** the type of the supplementary sequence (STIR, PSIR or MP2RAGE)

- **Target label:** 1 if the predicted lesion overlapped with a ground-truth lesion with IoU > 0.2, and the label was 0 otherwise.

## 2.4 Instance Classification Model

We trained a machine learning model to classify whether a given predicted lesion would be a true positive or false positive, using the above features. We experimented with several different machine learning models, including decision trees (DT), logistic regression (LR), k-nearest neighbours (kNN), support vector machine (SVM), random forest (RF), and XGBoost (XGB). These were implemented with the scikit-learn (v1.6.1) [4] and xgboost (v3.0.0) packages in Python. For kNN, LR, and SVC, the features were first standardised to mean 0 and standard deviation 1. A logit transformation was applied to the 'predicted scores' features for logistic regression to reverse the effect of the softmax/sigmoid. The 100 cases were split into five folds for cross validation, stratifying the splits to ensure a similar proportion of positive and negative examples and of different sequence representations across the folds, and also ensuring that observations associated with a given case were placed in only one fold. Average precision (AP) was used to select the best hyperparameters and the best model for ease of calculation. XGBoost gave the highest cross-validation performance, followed closely by random forest. Averaging the predictions from these two types of model slightly improved performance, so we used both. The best hyperparameters are presented in Table 1.

<table><caption>Table 1. Best hyperparameters found during search.</caption><thead><tr><th>RandomForestClassifier</th><th>XGBClassifier</th></tr></thead><tbody><tr><td>n_estimators = 200</td><td>n_estimators = 300</td></tr><tr><td>max_depth = 30</td><td>max_depth = 4</td></tr><tr><td>min Samples Split = 8</td><td>min_child_weight = 4</td></tr><tr><td>min Samples Leaf = 4</td><td>colsample_bytree = 0.3</td></tr><tr><td>max_features = log2</td><td>subsample = 0.5</td></tr><tr><td></td><td>learning_rate = 0.008</td></tr></tbody></table>
The final predictions in the challenge submission are based on the following:

1. A specific set of base masks (defining instances) is used for each subgroup of cases (i.e., those with STIR, PSIR, or MP2RAGE). See Table 2.

2. The instance probability is given by the average over the predictions from the five XGBoost models and the five random forest models.

3. There are cases in the challenge test set with both MP2RAGE and STIR. We use the MP2RAGE as the supplementary acquisition if a predicted instance is inside the MP2RAGE coverage, and otherwise we use the STIR.

<table><caption>**Table 2.** The mask type used for the case groups (based on the supplementary acquisition available). The instances were defined as the 3D connected components in these masks, and then the instance probabilities were based on predictions from the instance classification model.</caption><thead><tr><th>Group</th><th>Base prediction</th><th>Threshold</th><th>Test-time augmentation</th></tr></thead><tbody><tr><td>STIR</td><td>T2</td><td>10<sup>-5</sup></td><td>✓</td></tr><tr><td>PSIR</td><td>T2</td><td>10<sup>-6</sup></td><td>✓</td></tr><tr><td>MP2RAGE</td><td>mean</td><td>10<sup>-4</sup></td><td>✓</td></tr></tbody></table>

## 2.5 Results

The results presented here are the average cross-validation results, after obtaining probability predictions for each predicted lesion instance in the held-out test folds, and then running the FROC evaluation on the entire dataset of 100 cases.

<table><caption>**Table 3.** Performance of selected methods. The threshold reported here is the one giving the highest result for that method. TTA=test-time augmentation, XGB=XGBoost, RF=Random Forest.</caption><thead><tr><th>Method</th><th>Mask Type</th><th>FROC</th></tr></thead><tbody><tr><td>Raw preds</td><td>T2 mask at thr=10<sup>-6</sup></td><td>0.557</td></tr><tr><td>Raw preds</td><td>Mean mask at thr=0.001</td><td>0.572</td></tr><tr><td>Raw preds with TTA</td><td>Mean mask at thr=0.01</td><td>0.602</td></tr><tr><td>Postprocessed with XGB preds</td><td>T2 mask at thr=10<sup>-6</sup></td><td>0.633</td></tr><tr><td>Postprocessed with XGB + RF preds</td><td>Optimal mask chosen for each group (i.e., cases with MP2RAGE, PSIR, or STIR)</td><td><strong>0.655</strong></td></tr></tbody></table>
References

1. Benveniste, P.L., Lee, L.E., Prat, A., Vavasour, Z., Tam, R., Traboulsee, A., Kolind, S., Oh, J., Chen, M., Tsagkas, C., Granziera, C., Medina, N.L., Muhlau, M., Kirschke, J., McGinnis, J., Reich, D., Hemond, C., Callot, V., Demortière, S., Au-doin, B., Nair, G., Filippi, M., Valsasina, P., Rocca, M., Ciccarelli, O., Yiannakas, M., Granberg, T., Ouellette, R., Shahamat, T., Bakshi, R., Mainero, C., Treaba, C., Kerbrat, A., Bannier, E., Edan, G., Labauge, P., O'Grady, K., Smith, S., Shepherd, T., Charlson, E., Brisset, J.C., Talbott, J., Liu, Y., Lombaert, H., Cohen-Adad, J.: Reinforcing the generalizability of spinal cord multiple sclerosis lesion segmentation models. In: 41st Annual Scientific Meeting of the European Society for Magnetic Resonance in Medicine and Biology (ESMRMB) (2025)

2. De Leener, B., Lévy, S., Dupont, S.M., Fonov, V.S., Stikov, N., Louis Collins, D., Callot, V., Cohen-Adad, J.: SCT: Spinal Cord Toolbox, an open-source software for processing spinal cord MRI data. NeuroImage 145(Pt A) (Jan 2017). https://doi.org/10.1016/j.neuroimage.2016.10.009

3. Lodé, B., Hussein, B.R., Meurée, C., Walsh, R., Gaubert, M., Lassalle, N., Courbon, G., Martin, A., Le Bars, J., Durand-Dubief, F., Bourre, B., Maarouf, A., Outteryck, O., Mehier, C., Poulin, A., Cathelineau, C., Hong, J., Criton, G., Motillon-Alonso, S., Lecler, A., Charbonneau, F., Duron, L., Bani-Sadr, A., Delpierre, C., Ferré, J.C., Edan, G., Cotton, F., Casey, R., Galassi, F., Combès, B., Kerbrat, A.: Evaluation of a deep learning segmentation tool to help detect spinal cord lesions from combined T2 and STIR acquisitions in people with multiple sclerosis. European Radiology (Apr 2025). https://doi.org/10.1007/s00330-025-11541-0

4. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V.: Scikit-learn: Machine learning in Python. The Journal of Machine Learning Research (JLMR) 12 (2011)

5. Walsh, R., Gaubert, M., Meurée, C., Hussein, B.R., Kerbrat, A., Casey, R., Combès, B., Galassi, F.: Multi-sequence learning for multiple sclerosis lesion segmentation in spinal cord MRI. In: MICCAI 2024 - 27th International Conference on Medical Image Computing and Computer-Assisted Intervention. Marrakech, Morocco (Oct 2024)

6. Walsh, R., Meuree, C., Kerbrat, A., Masson, A., Hussein, B.R., Gaubert, M., Galassi, F., Combes, B.: Expert Variability and Deep Learning Performance in Spinal Cord Lesion Segmentation for Multiple Sclerosis Patients. In: 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS) (Jun 2023). https://doi.org/10.1109/CBMS58004.2023.00263
Modular Cross-Attention Fusion for Multi-Modal
MS Spine Lesion Segmentation with Missing
Modalities

Jiacheng Wang<sup>1,2</sup>, Han Liu<sup>2</sup>, Hao Li<sup>1</sup>, Francesca Bagnato<sup>3</sup>, and Ipek Oguz<sup>1</sup>

¹ Vanderbilt University, Nashville, TN, USA

² Siemens Healthineers, Princeton, NJ, USA

³ Vanderbilt University Medical Center, Nashville, TN, USA

jiacheng.wang.1@vanderbilt.edu, han.liu@siemens-healthineers.com,
francesca.r.bagnato@vumc.org, ipek.oguz@vanderbilt.edu

**Abstract.** We present a novel modular fusion architecture for multiple sclerosis (MS) spine lesion segmentation that handles arbitrary combinations of missing MRI modalities. Our approach employs a cross-attention mechanism inspired by prompt encoding in vision foundation models, enabling robust performance on unseen modality combinations during inference. The method treats T2-weighted imaging as the primary modality and encodes auxiliary modalities (STIR, PSIR, MP2RAGE) as contextual prompts through cross-attention fusion. We combine this with an ensemble approach using modality-dropout training to further enhance missing modality robustness. The proposed architecture demonstrates the capability to leverage additional information from available modalities while maintaining performance when modalities are absent.

**Keywords:** Multiple Sclerosis · Missing Modalities · Cross-Attention · Multi-Modal Fusion · Medical Image Segmentation.

# 1 Introduction

Multiple sclerosis (MS) spine lesion segmentation from MRI presents unique challenges due to varying scanning protocols. While T2-weighted sagittal images are consistently acquired, auxiliary sequences such as STIR, PSIR, and MP2RAGE may be missing depending on the scanning protocol, equipment availability, or clinical workflow. This variability creates a practical missing modality problem that existing segmentation methods struggle to address effectively.

Traditional multi-modal approaches either require all modalities to be present or use simple concatenation strategies that fail to generalize to unseen modality combinations [4,5]. Recent advances in vision foundation models [7], particularly the prompt-based attention mechanisms in SAM [3], suggest alternative approaches for handling variable input configurations.

We propose a modular cross-attention fusion architecture [1] that treats auxiliary modalities as contextual prompts, enabling robust inference on arbitrary modality combinations, including those never encountered during training.
# 2 Methodology

## 2.1 Modular Fusion Architecture

As shown in Fig. 1, our architecture consists of two primary components: (1) a T2w primary encoder that processes the mandatory T2-weighted images, and (2) a modular auxiliary encoder that processes available complementary modalities through cross-attention fusion.

**T2w Primary Branch:** The T2w primary branch $\mathcal{E}_{T2}$ follows the standard nnU-Net architecture [2]:

$$ \mathbf{F}_{T2} = \mathcal{E}_{T2}(\mathbf{I}_{T2}) \qquad (1) $$

where $\mathbf{I}_{T2} \in \mathbb{R}^{H \times W \times D}$ is the T2-weighted input volume and $\mathbf{F}_{T2} \in \mathbb{R}^{C \times H' \times W' \times D'}$ represents the extracted feature maps at multiple resolution levels.

**Auxiliary Modality Encoding:** For each available auxiliary modality $m \in \{\text{STIR, PSIR, MP2RAGE}\}$, we extract features using modality-specific encoders:

$$ \mathbf{F}_{m} = \mathcal{E}_{m}(\mathbf{I}_{m}) \qquad (2) $$

where $\mathbf{I}_{m}$ represents the input volume for modality $m$. When modality $m$ is missing, we set $\mathbf{I}_{m} = \mathbf{0}$ (zero-padding).

**Cross-Attention Fusion:** Inspired by the prompt mechanism in SAM, we treat auxiliary modality features as contextual prompts that guide the primary T2w segmentation through cross-attention [6]. The fusion operation is defined as:

$$ \mathbf{F}_{\text{fused}} = \mathcal{F}_{\text{cross-attn}}(\mathbf{F}_{T2}, \{\mathbf{F}_{m}\}_{m=1}^{M}) \qquad (3) $$

The cross-attention mechanism computes attention weights between T2w features (queries) and auxiliary features (keys/values):

$$ \mathbf{Q} = \mathbf{F}_{T2} \mathbf{W}_{Q} \qquad (4) $$

$$ \mathbf{K} = \text{Concat}(\{\mathbf{F}_{m} \mathbf{W}_{K,m}\}_{m=1}^{M}) \qquad (5) $$

$$ \mathbf{V} = \text{Concat}(\{\mathbf{F}_{m} \mathbf{W}_{V,m}\}_{m=1}^{M}) \qquad (6) $$

$$ \mathbf{A} = \text{Softmax} \left( \frac{\mathbf{Q} \mathbf{K}^{T}}{\sqrt{d_k}} \right) \qquad (7) $$

$$ \mathbf{F}_{\text{fused}} = \mathbf{F}_{T2} + \mathbf{A} \mathbf{V} \qquad (8) $$

where $\mathbf{W}_{Q}, \mathbf{W}_{K,m}$, and $\mathbf{W}_{V,m}$ are learned projection matrices, and $d_k$ is the key dimension. The T2 features serve as queries while auxiliary modality features provide keys and values, resulting in $\mathbf{F}_{\text{fused}}$ that enhances the primary T2 representation with relevant information from available modalities.
<figure><img src="image_5.png" /><figcaption>Fig. 1. Overview of the modular cross-attention fusion architecture. The T2 primary branch processes mandatory T2-weighted images, while auxiliary modalities (STIR, PSIR, MP2RAGE) are encoded as contextual prompts through cross-attention fusion. The example shows a case with T2+STIR where PSIR and MP2RAGE are missing and handled via zero-padding without architectural modifications.</figcaption></figure>

## 2.2 Missing Modality Handling

One of the key innovations of our proposed solution lies in how our architecture handles missing modalities. When modality *m* is absent:

$$F_m = \mathcal{E}_m(\mathbf{0}) = \mathbf{0}_{\text{features}} \qquad (9)$$

The cross-attention mechanism naturally adapts to this scenario:

$$\text{Attention}(\mathbf{Q}, \mathbf{K}_{\text{available}}, \mathbf{V}_{\text{available}}) = \text{Softmax} \left( \frac{\mathbf{Q}\mathbf{K}_{\text{available}}^T}{\sqrt{d_k}} \right) \mathbf{V}_{\text{available}} \quad (10)$$

This formulation allows the model to dynamically weight available modalities without architectural modifications during inference.

## 2.3 Ensemble Strategy with Modality Dropout

To further enhance robustness, we employ an ensemble approach combining two complementary strategies:

**ModalityPrompt Trainer:** We use a modality-dropout strategy where during training, we set complete modalities to zero with probability for the incomplete cases

$$\mathbf{I}_{m}^{\text{train}} = \begin{cases} \mathbf{I}_{m} & \text{present modality} \\ \mathbf{0} & \text{missing modality} \end{cases} \qquad (11)$$

This forces the model to learn representations that are robust to missing modalities.
**ModularFusion Trainer:** We use the cross-attention fusion approach described above to train on all available modality combinations.

**Ensemble Inference:** Final predictions combine both strategies through probability averaging:

$$ \mathbf{P}_{final} = \frac{1}{6} \left( \sum_{k=0}^{4} \mathbf{P}_{\text{ModalityPrompt}}^{(k)} + \mathbf{P}_{\text{ModularFusion}} \right) \quad (12) $$

where k represents the 5-fold cross-validation models from ModalityPrompt training.

## 2.4 Training Loss Function

The model is trained using a combination of Dice loss and Cross-Entropy loss:

$$ \mathcal{L} = \mathcal{L}_{\text{Dice}} + \mathcal{L}_{\text{CE}} = - \frac{2|\mathbf{Y} \cap \hat{\mathbf{Y}}|}{|\mathbf{Y}| + |\hat{\mathbf{Y}}|} - \sum_{i} y_i \log(\hat{y}_i) \quad (13) $$

where Y and ˆY represent ground truth and predicted segmentation masks, respectively.

# 3 Key Innovations

**Generalization to Unseen Combinations** The cross-attention mechanism enables inference on modality combinations never seen during training. For example, a model trained on (T2, STIR), (T2, PSIR), and (T2, MP2RAGE) combinations can effectively process (T2, STIR, MP2RAGE) inputs by leveraging learned attention patterns, even though this three-modality combination was never encountered during training.

**Prompt-Inspired Design** By treating auxiliary modalities as contextual prompts rather than equal partners in concatenation-based fusion, the architecture maintains the primacy of T2 imaging while adaptively incorporating available complementary information.

**Zero-Shot Missing Modality Handling** The architecture requires no retraining or architectural modification when encountering new missing modality patterns, enabling deployment across diverse clinical environments.

# 4 Results

Figure 2 presents qualitative results across different modality combinations. Our ensemble approach shows improved lesion boundary delineation and reduced
<figure><img src="image_6.png" /><figcaption>**Fig. 2.** Qualitative comparison showing segmentation results for different modality combinations. Row 1: T2+STIR case. Row 2: T2+PSIR case. Row 3: T2+PSIR+STIR case. Columns show: Ground Truth, Our Ensemble Result, and Baseline (Modality Dropout). Our method shows improved lesion boundary delineation and reduced false positives, particularly in challenging cases with subtle lesions.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 3.** Free Response Operating Characteristic (FROC) curves comparing our ensemble approach with baseline methods. The curve shows sensitivity versus false positives per image, demonstrating superior detection performance across different operating points. Our ensemble method (red line) consistently outperforms both the baseline modality dropout approach and individual components.</figcaption></figure>
false positives compared to the baseline modality dropout method. The cross-attention mechanism effectively leverages complementary information from available auxiliary modalities, leading to more accurate segmentations particularly in cases with subtle lesions.

Figure 3 shows the FROC curves for different methods. Our ensemble approach demonstrates consistently higher sensitivity across the full range of false positive rates, indicating robust lesion detection capabilities. The improvement is particularly pronounced at lower false positive rates, which is clinically relevant for reducing unnecessary follow-up procedures.

## 5 Implementation details (nnU-Net v2 with modular fusion)

**Base architecture.** We adopt the nnU-Net v2 3D full-resolution configuration (3d_fullres) as the base U-Net family. To accommodate multi-modal inputs, the base network is wrapped by a *modular fusion* layer that fuses up to four modalities (T2, STIR, PSIR, MP2RAGE) before feeding features into the U-Net. We use the default *attention*-based fusion with a fusion width of 32 channels and enable deep supervision across decoder scales. The U-Net depth (#levels), kernel sizes, and channel multipliers follow nnU-Net's automatically generated plans for the dataset (3D full resolution).

**Input data and preprocessing.** Inference strictly uses the *preprocessed-and-registered* volumes. For each case, we locate the *preprocessedAndRegistered* T2 as the reference space and use it to assemble the four-channel nnU-Net input. If a modality is unavailable, we insert a zero-filled image in the same space to keep channel consistency. No additional preprocessing beyond the challenge's provided “preprocessed & registered” data is applied during inference.

**Losses (cost function).** Training uses the standard nnU-Net composite loss $\mathcal{L} = \lambda_{\text{CE}} \mathcal{L}_{\text{CE}} + \lambda_{\text{Dice}} \mathcal{L}_{\text{Dice}}$, with cross-entropy and a memory-efficient soft Dice term. Background is excluded in the Dice ($\text{do_bg=false}$); Dice smoothing $\epsilon = 10^{-5}$. When deep supervision is enabled, auxiliary outputs are weighted by $w_i \propto 2^{-i}$ with the lowest-resolution head disabled ($w_{\text{lowest}} = 0$) and weights re-normalized to sum to one. In the optional two-stage refinement setting, the total loss is a convex combination of initial and refined predictions: $\mathcal{L}_{\text{total}} = 0.3 \mathcal{L}_{\text{init}} + 0.7 \mathcal{L}_{\text{refine}}$.

**Optimization scheme.** Mixed-precision training (automatic casting) is used together with gradient scaling and gradient clipping (global norm 12). We keep nnU-Net's default optimizer, learning-rate schedule, data augmentation, and early-stopping/checkpointing policies as provided by nnU-Net v2 (no changes). At inference, we apply test-time mirroring and Gaussian importance weighting with a tiling step of 0.5.

**Training schedule and model selection.** We train using nnU-Net's default epoch/iteration schedule for the selected configuration and dataset split. The model used for evaluation is the checkpoint_best.pth produced by nnU-Net's
internal validation during training ("best" according to the default nnU-Net criterion).

**Postprocessing from softmax to instances & probabilities.** During inference, we export both the hard segmentation and the per-class softmax probabilities. From the foreground (lesion) probability volume $p(\mathbf{x})$ we create a binary map by thresholding (default $\tau = 0.5$; we additionally report sensitivity to $\tau \in \{0.3, 0.4, 0.5\}$). We then perform 3D connected-component labeling to obtain instance masks. Tiny components (< 15 voxels) are removed to suppress noise. For each surviving instance, we report a confidence score defined as the *maximum* voxel-wise foreground probability within that connected component; we also compute the mean and 95th-percentile for analysis (not used for ranking). Outputs are saved as: (i) an instance-labeled NIfTI mask aligned to the preprocessed-and-registered T2 reference, and (ii) a CSV file mapping *label* → *p* (instance probability).

# 6 Conclusion

We present a novel modular cross-attention fusion approach for MS spine lesion segmentation that addresses the practical challenge of missing modalities in clinical settings. The prompt-inspired design enables robust performance on arbitrary modality combinations, including those unseen during training. Combined with ensemble strategies using modality dropout, the approach provides a comprehensive solution for multi-modal medical image segmentation under realistic clinical constraints.

The methodology demonstrates the potential of attention-based fusion mechanisms borrowed from foundation models for addressing domain-specific challenges in medical imaging, particularly the ubiquitous problem of missing modalities in clinical studies.

# References

1. Havaei, M., Guizard, N., Chapados, N., Bengio, Y.: Hemis: Hetero-modal image segmentation. In: Medical Image Computing and Computer-Assisted Intervention (MICCAI). Lecture Notes in Computer Science, vol. 9901, pp. 469–477. Springer (2016)

2. Isensee, F., Jaeger, P., Kohl, S., et al.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18(2), 203–211 (2021)

3. Kirillov, A., Mintun, E., Ravi, N., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4015–4026 (2023)

4. Liu, H., Fan, Y., Xu, Z., Dawant, B.M., Oguz, I.: Learning site-specific styles for multi-institutional unsupervised cross-modality domain adaptation. In: International Challenge on Cross-Modality Domain Adaptation for Medical Image Segmentation, pp. 372–385. Springer (2023)
8

J. Wang et al.

5. Neverova, N., Wolf, C., Taylor, G., Nebout, F.: Moddrop: adaptive multi-modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(8), 1692–1706 (2015)

6. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems (NeurIPS). pp. 5998–6008 (2017)

7. Wang, J., Li, H., Liu, H., Hu, D., Lu, D., Yoon, K., Barter, K., Bagnato, F., Oguz, I.: Ssl2 self-supervised learning meets semi-supervised learning: multiple clerosis segmentation in 7t-mri from large-scale 3t-mri. In: Medical Imaging 2023: Image Processing. vol. 12464, pp. 134–144. SPIE (2023)