# Fine-tuning Segment Anything for Accurate Tumor Tracking in MRI Sequences

Valentin Boussot, Cédric Hémon, Jean-Claude Nunes, Jean-Louis Dillenseger

► To cite this version:

Valentin Boussot, Cédric Hémon, Jean-Claude Nunes, Jean-Louis Dillenseger. Fine-tuning Segment Anything for Accurate Tumor Tracking in MRI Sequences. TrackRAD2025 Challenge, Real-time Tumor Tracking for MRI-guided Radiotherapy, MICCAI 2025, Sep 2025, Daejeon, South Korea. hal-05334604

HAL Id: hal-05334604

https://hal.science/hal-05334604v1

Submitted on 28 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Fine-tuning Segment Anything for Accurate
Tumor Tracking in MRI Sequences

Valentin Boussot¹[0009-0003-2465-5458], Cédric Hémon¹[0009-0003-6669-5108],
Jean-Claude Nunes¹, and Jean-Louis Dillenseger¹

Univ Rennes 1, CLCC Eugène Marquis, INSERM, LTSI - UMR 1099, F-35000
Rennes, France
https://ltsi.univ-rennes.fr/

**Abstract.** In this work, we address the TrackRAD2025 challenge of real-time tumor tracking in cine-MRI sequences of the thoracic and abdominal regions under strong data scarcity constraints. Two complementary strategies were explored: (i) unsupervised registration with the IMPACT similarity metric and (ii) foundation model-based segmentation leveraging SAM 2.1 and its recent variants through prompt-based interaction. Due to the one-second runtime constraint, the SAM-based method was ultimately selected.

The final configuration used SAM2.1b+ with mask-based prompts from
the first annotated slice, fine-tuned solely on the small labeled subset
from TrackRAD2025. Training was configured to minimize overfitting,
using 1024 x 1024 patches (batch size 1), standard augmentations, and
a balanced Dice + IoU loss. A low uniform learning rate (1 x 10⁻⁵)
was applied to all modules (prompt encoder, decoder, Hiera backbone)
to preserve generalization while adapting to annotator-specific styles.
Training lasted 300 epochs ( 12h on RTX A6000, 48GB).

The same inference strategy was consistently applied across all anatom-
ical sites and MRI field strengths. Test-time augmentation was consid-
ered but ultimately discarded due to negligible performance gains. The
final model was selected based on the highest Dice Similarity Coefficient
achieved on the validation set after fine-tuning.

# 1 Introduction

Medical image segmentation and registration play a crucial role in numerous
clinical workflows, enabling accurate delineation of anatomical structures and
pathological regions for diagnosis, treatment planning, and follow-up [3,7]. In the
context of the TrackRAD2025 challenge [8], the task involves cine-MRI sequences
of the thoracic and abdomen region, where both spatial accuracy and temporal
consistency are essential for robust tracking.

A key constraint in this challenge is the scarcity of annotated training data,
which limits the applicability of conventional supervised deep learning methods
that typically rely on large-scale, fully annotated datasets to reach state-of-the-
art performance [5].
Given these constraints, we adopted a dual methodological strategy devel-
oped using KonfAI [1]:

- An unsupervised registration approach based on IMPACT [2], which lever-
ages deformable image registration to propagate the initial mask across sub-
sequent frames in the temporal sequences without requiring additional man-
ual annotations.

- The use of foundation segmentation models, such as SAM [4] (Segment Anything Model) and MedSAM [6], which are trained on large and diverse datasets and can generalize to novel imaging modalities and anatomical sites.

The unsupervised nature of the registration approach eliminates the need for model training and parameter tuning on the challenge dataset, making it inherently robust to data scarcity. In parallel, the use of foundation models [5] provides a strong prior for anatomical segmentation, with the possibility of refining their performance through fine-tuning on the small annotated subset available.

This hybrid strategy is designed to ensure accurate and generalizable tumor
tracking in cine-MRI, even in the face of data scarcity.

2 Methods

2.1 Data

This study exclusively relied on the annotated subset of the TrackRAD2025 dataset for both the preliminary validation of the proposed method and the fine-tuning of the network. No additional public datasets were employed.

## 2.2 Model

Unsupervised Registration with IMPACT

IMPACT is a similarity metric specifically designed for multimodal medical im-
age registration. It leverages high-level anatomical features extracted from large,
pretrained segmentation models to guide the registration process. By integrating
these semantic representations, IMPACT mitigates the limitations of traditional
intensity-based similarity measures, which are often sensitive to noise, artifacts,
and modality-specific intensity variations.

IMPACT was integrated into the Elastix registration framework, enabling deformable image registration between frames in a cine-MRI sequence. This approach aimed to propagate the initial mask provided for the first slice to all subsequent frames via spatial transformations estimated through registration.

While theoretically well-suited to the problem, the method faced practical
constraints: the challenge imposes a one-second prediction time limit per case,
which required a drastic reduction in the number of optimization iterations. De-
spite parameter tuning, this time constraint significantly degraded the achievable
registration accuracy, preventing the full exploitation of IMPACT's potential.
Accordingly, we focus on the second approach, which aligns better with the
challenge's runtime constraints.
## Tracking with SAM

For the use of segmentation foundation models, we tested many of these models that have been validated on the available annotated dataset. The SAM, Med-SAM2 models and other variants were tested on the validation dataset. We then tested several prompting techniques (point sample, box, mask of the first slice) and kept the last method using the first slice. The use of test-time augmentation was also tested but did not yield any significant improvement. The same method was used for all anatomical locations and different MRI field strengths. The average inference time per cine MRI frame was approximately 10 milliseconds on our hardware.

The model SAM2.1 b+ was then fine-tuned as explained in the following section.

## 2.3 Training

During initial experiments using SAM for inference, we observed that the predictions were generally accurate. However, manual annotations consistently overestimated the segmented tumor volumes compared to SAM outputs. To account for this annotator bias, we fine-tuned the model on the labeled subset, enabling it to learn this discrepancy and align its predictions with the provided ground truth masks.

Fine-tuning was performed following the specific training requirements of the SAM architecture. Input patches of size 1024×1024 were extracted from the cine MRI sequences. Standard data augmentation techniques were applied, including RandomHorizontalFlip, RandomAffine, ColorJitter, and RandomGrayscale, to improve generalization and robustness. The model was trained with a batch size of 1 and supervised using a composite loss combining Dice loss (weight=1) and IoU loss (weight=1). Optimization was performed using the AdamW optimizer. Sequences of 8 consecutive frames (number of frames = 8) were provided as input.

The training procedure employed binary masks as prompts, with a learning rate of 0.0005 for the prompt network, the decoder, and the Hiera-based encoder. This relatively low learning rate was selected to preserve the pretrained capabilities of SAM while enabling it to adapt to the mask-prompt bias present in the training data.

The model was trained for 300 epochs, and the network from the final epoch was selected as the deployed model for subsequent evaluations. Training required approximately 12 hours in total on an NVIDIA RTX A6000 GPU (48 GB).

## 2.4 Evaluation

To assess the quality of tumor tracking in the TrackRAD2025 challenge, we employed the full set of evaluation metrics defined by the challenge organizers. These metrics aim to quantify both segmentation fidelity and the geometric accuracy of temporal tracking. They include:
- **Dice Similarity Coefficient (DSC):** Measures the spatial overlap between the predicted and reference masks, defined as twice the intersection divided by the sum of their volumes:

$$DSC = \frac{2|A \cap B|}{|A| + |B|}$$

A higher DSC (maximum value of 1) indicates stronger agreement. This metric is widely used in medical image segmentation.

- **Hausdorff Distance (HD) and Average Surface Distance (ASD):** These metrics evaluate the distance between the surfaces of the predicted and reference masks. HD measures the largest surface-to-surface distance, while ASD reports the average distance, providing a more stable indicator of contour accuracy.

- **Center of Mass Distance (CD):** The Euclidean distance between the centroids of the predicted and reference segmentations. This captures the spatial consistency of tracking over time.

- **Inference Speed:** Given that real-time performance is a critical requirement in this challenge, the inference speed of the method is recorded. The competition imposes a maximum prediction time per case (approximately one second), making this a decisive performance criterion.

- **Estimated Radiation Dose Delivery Accuracy:** Although less common in traditional segmentation benchmarks, this metric quantifies the clinical impact of segmentation on the ability to deliver the prescribed radiotherapy dose accurately. It is included in the challenge to reflect the ultimate clinical goal of real-time treatment adaptation.

This combination of classical segmentation metrics (DSC, HD, ASD), temporal tracking measures (CD, inference speed), and clinical impact assessment (dose delivery accuracy) provides a comprehensive framework for evaluating real-time tumor tracking methods.

# 3 Results

We evaluated the proposed method on the TrackRAD2025 validation set using the metrics described in Section 2.4, excluding dose-related metrics.

Table 1 reports the performance of different foundation model configurations. The best-performing baseline was obtained with the SAM 2.1.1 configuration, which achieved a DSC of 0.916, a Hausdorff Distance of 3.26 mm, an Average Surface Distance of 1.20 mm, and a Center of Mass Distance of 1.34 mm. MedSAM-based variants performed worse across all metrics, particularly in terms of distance measures. Although pretrained SAM models already yielded strong segmentation results, a systematic overestimation of tumor volume was observed in the manual annotations compared to the model predictions. This discrepancy motivated the use of fine-tuning to adapt the model to the annotator style and improve alignment with the provided ground truth. We also
evaluated an ensemble prompt strategy on the SAM2.1b+ model, combining di- lated and eroded versions of the initial mask to enhance robustness. However, this approach did not outperform the best model and slightly degraded perfor- mance in some metrics.

<table><caption>**Table 1.** Comparison of different prompted foundation models for segmentation on the validation set. Best values per metric are in bold.</caption><thead><tr><th>Model</th><th>DSC ↑</th><th>HD [mm]</th><th>↓ ASD [mm]</th><th>↓ CD [mm]</th><th>↓</th></tr></thead><tbody><tr><td>Baseline</td><td>0.770</td><td>8.27</td><td>3.88</td><td>6.42</td><td></td></tr><tr><td>SAM 2.1 t</td><td>0.911</td><td>3.45</td><td>1.27</td><td>1.51</td><td></td></tr><tr><td>SAM 2.1 b+</td><td>0.915</td><td>3.41</td><td>1.29</td><td>1.46</td><td></td></tr><tr><td><strong>SAM 2.1 l</strong></td><td><strong>0.916</strong></td><td><strong>3.26</strong></td><td><strong>1.20</strong></td><td><strong>1.34</strong></td><td></td></tr><tr><td>MedSAM2</td><td>0.869</td><td>19.76</td><td>17.01</td><td>17.28</td><td></td></tr><tr><td>MedSAM2 liver</td><td>0.842</td><td>29.62</td><td>27.32</td><td>27.51</td><td></td></tr><tr><td>Ensemble prompt</td><td>0.908</td><td>3.51</td><td>1.29</td><td>1.39</td><td></td></tr></tbody></table>

We then investigated the effect of fine-tuning SAM2.1l to better match the annotator style. As shown in Table 2, the evaluation on the training subset showed consistent improvements across all metrics, with DSC increasing from 0.916 to 0.929 and HD decreasing from 3.26mm to 2.78mm. These gains, while expected on the training set, indicate that the model successfully adapted to the annotation style without signs of overfitting.

<table><caption>**Table 2.** Performance before and after fine-tuning on the labeled subset. Best values per metric are in bold.</caption><thead><tr><th>Model</th><th>DSC ↑</th><th>HD [mm]</th><th>↓ ASD [mm]</th><th>↓ CD [mm]</th><th>↓</th></tr></thead><tbody><tr><td>SAM 2.1 b+</td><td>0.915</td><td>3.41</td><td>1.29</td><td>1.46</td><td></td></tr><tr><td><strong>Fine-tuned SAM 2.1 b+</strong></td><td><strong>0.929</strong></td><td><strong>2.78</strong></td><td><strong>1.00</strong></td><td><strong>1.16</strong></td><td></td></tr></tbody></table>

## 3.1 Preliminary test set evaluation

To assess the generalization ability of the fine-tuned model, we further eval- uated both versions (with and without fine-tuning) on the preliminary unseen test set provided by the organizers. As shown in Table 3, fine-tuning consistently improved all metrics, including DSC, center of mass accuracy, and shape con- sistency, confirming that the model adapted well to the annotator bias without overfitting.

## 3.2 Official Challenge Results

Our final submission achieved the **6th best overall position** on the official TrackRAD2025 leaderboard, with competitive performance in several key met- rics:
<table><caption>**Table 3.** Preliminary test set evaluation. Fine-tuning significantly improved performance across all metrics.</caption><thead><tr><th>Rank</th><th>Model</th><th>DSC ↑</th><th>CD [mm]</th><th>↓ Rel. D98 ↑</th><th>SD95 [mm]</th><th>↓ SDavg [mm]</th><th>↓</th></tr></thead><tbody><tr><td>3rd</td><td>SAM 2.1 b+ (fine-tuned)</td><td><strong>0.9152</strong></td><td><strong>1.4912</strong></td><td><strong>0.9803</strong></td><td>4.2141</td><td>1.5008</td><td></td></tr><tr><td>5th</td><td>SAM 2.1 b+ (no fine-tune)</td><td>0.9039</td><td>1.8864</td><td>0.9312</td><td>4.9406</td><td>1.7000</td><td></td></tr></tbody></table>

<table><caption>**Table 4.** Final performance on the hidden TrackRAD2025 test set. Rankings per metric are in parentheses.</caption><thead><tr><th>Rank</th><th>Team</th><th>DSC ↑</th><th>CD [mm]</th><th>↓ Rel. D98 ↑</th><th>SD95 [mm]</th><th>↓ SDavg [mm]</th><th>↓</th></tr></thead><tbody><tr><td>1<sup>st</sup></td><td>Track'n'Treat</td><td><strong>0.8909</strong> (1)</td><td><strong>1.4733</strong> (1)</td><td>0.9361 (4)</td><td><strong>4.2171</strong> (1)</td><td><strong>1.6642</strong> (1)</td><td></td></tr><tr><td>2<sup>nd</sup></td><td>Hkini_Team</td><td>0.8860 (2)</td><td>1.4858 (2)</td><td><strong>0.9628</strong> (1)</td><td>4.4969 (2)</td><td>1.8565 (2)</td><td></td></tr><tr><td>3<sup>rd</sup></td><td>felixknispelrwth</td><td>0.8764 (5)</td><td>1.6576 (3)</td><td>0.9250 (7)</td><td>4.5409 (3)</td><td>1.9174 (3)</td><td></td></tr><tr><td>4<sup>th</sup></td><td>CIT</td><td>0.8712 (6)</td><td>1.9564 (4)</td><td>0.9392 (2)</td><td>5.2583 (4)</td><td>2.1903 (4)</td><td></td></tr><tr><td>5<sup>th</sup></td><td>Reg'n'Track</td><td>0.8801 (3)</td><td>2.1031 (5)</td><td>0.9321 (5)</td><td>5.6407 (7)</td><td>2.2416 (5)</td><td></td></tr><tr><td>6<sup>th</sup></td><td>BreizhTrack (ours)</td><td>0.8794 (4)</td><td>2.3192 (7)</td><td>0.9363 (3)</td><td>5.6298 (6)</td><td>2.6091 (8)</td><td></td></tr></tbody></table>

Our final submission ranked 6th overall on the TrackRAD2025 leaderboard (Table 4). The top teams exhibited very similar performance, with differences of only 0.01–0.02 in Dice and small variations in CD and SD metrics, illustrating the highly competitive nature of the challenge.

## 4 Discussion

The results achieved by our approach place it among the top-performing methods in the TrackRAD2025 challenge, underscoring the promising potential of foundation models pretrained on large-scale natural image datasets. Despite the inherent domain gap between natural and medical images, these models consistently demonstrate strong generalization capabilities. Notably, the differences in performance among the leading teams were very small, highlighting how competitive and tightly contested the challenge was.

Our method reached Dice scores above 0.91 on both the training set (50 annotated patients) and the preliminary test set (8 unseen patients), indicating good generalization during validation. However, on the final hidden test set, the Dice dropped to 0.8794, revealing a notable performance gap. Interestingly, several teams with high preliminary scores experienced similar drops, while others with much lower preliminary performance ranked higher in the final test. This suggests that generalizing across the full diversity of anatomical cases and acquisition protocols remains challenging, even for powerful foundation models.

Despite this, our method remained highly competitive, ranking 4th in Dice and 3rd in relative D98, while maintaining inference times well below the 1-second constraint.
As previously demonstrated in the IMPACT framework, there is a growing shift toward integrating foundation models into medical imaging pipelines. Their ability to extract high-level semantic features with minimal supervision makes them particularly well-suited for data-scarce clinical applications. Our findings reinforce this trend, showing that such models can be successfully adapted to complex medical tasks despite limited training data.

Finally, we observed that fine-tuning on a small number of annotated cases improved performance on unseen patients. However, it remains unclear whether this improvement reflects a genuine gain in segmentation accuracy or a model adaptation to specific annotation styles.

# 5 Author contributions

Conceptualization: V. Boussot, C. Hémon. Methodology: V. Boussot, C. Hémon. Software and experiments: V. Boussot. Supervision: J.-L. Dillenseger, J.-C. Nunes. Writing – original draft: V. Boussot, C. Hémon. Writing – review and editing: all authors.

## Code availability

The inference code used for our TrackRAD2025 submission is publicly available at: https://github.com/vboussot/Trackrad2025. We also release the fine-tuned weights of the SAM 2.1 b+ model, trained on the annotated subset of the TrackRAD2025 dataset, at: https://huggingface.co/VBoussot/Trackrad2025. These resources allow full reproduction of our inference pipeline and challenge results.

## Acknowledgment

The work presented in this article was supported by the Brittany Region through its Allocations de Recherche Doctorale framework and by the French National Research Agency as part of the VATSop project (ANR-20-CE19-0015). Additionally, it was supported by a PhD scholarship Grant from Elekta AB (C.Hémon). The authors have no relevant financial or non-financial interests to disclose. While preparing this work, the authors used ChatGPT to enhance the writing structure and refine grammar. After using these tools, the authors reviewed and edited the content as needed and took full responsibility for the publication's content.

## References

1. Boussot, V., Dillenseger, J.L.: Konfai: A modular and fully configurable framework for deep learning in medical imaging (2025). https://doi.org/10.48550/ARXIV.2508.09823
2. Boussot, V., Hémon, C., Nunes, J.C., Dowling, J., Rouzé, S., Lafond, C., Barateau, A., Dillenseger, J.L.: Impact: A generic semantic loss for multimodal medical image registration. arXiv preprint arXiv:2503.24121 (2025)

3. Chen, X., Wang, X., Zhang, K., Fung, K.M., Thai, T.C., Moore, K., Mannel, R.S., Liu, H., Zheng, B., Qiu, Y.: Recent advances and clinical applications of deep learning in medical image analysis. Medical Image Analysis **79**, 102444 (Jul 2022)

4. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)

5. Kolides, A., Nawaz, A., Rathor, A., Beeman, D., Hashmi, M., Fatima, S., Berdik, D., Al-Ayyoub, M., Jararweh, Y.: Artificial intelligence foundation and pre-trained models: Fundamentals, applications, opportunities, and social impacts. Simulation Modelling Practice and Theory **126**, 102754 (Jul 2023). https://doi.org/10.1016/j.simpat.2023.102754, https://www.sciencedirect.com/science/article/pii/S1569190X2300031X

6. Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications **15**, 654 (2024)

7. Viergever, M.A., Maintz, J.B.A., Klein, S., Murphy, K., Staring, M., Pluim, J.P.W.: A survey of medical image registration – under review. Medical Image Analysis **33**, 140–144 (Oct 2016)

8. Wang, Y., Lombardo, E., Thummerer, A., Blöcker, T., Fan, Y., Zhao, Y., Papadopoulou, C.I., Hurkmans, C., Tijssen, R.H., Görts, P.A., et al.: Trackrad2025 challenge dataset: real-time tumor tracking for mri-guided radiotherapy. Medical physics **52**(7), e17964 (2025)