Med-Tuning: A New Parameter-Efficient Tuning Framework for Medical Volumetric Segmentation

Authors: jiachen  shen
wenxuan  wang
chen  chen
jianbo  jiao
jing  liu
yan  zhang
shanshan  song
jiangyun  li
Venue: MIDL_io_2024_Conference
Published: 2024-01-25 07:24:11

Abstract: The "pre-training then fine-tuning (FT)" paradigm is widely adopted to boost the model performance of deep learning-based methods for medical volumetric segmentation. However, conventional full FT incurs high computational and memory costs. Thus, it is of increasing importance to fine-tune pre-trained models for medical volumetric segmentation tasks in a both effective and parameter-efficient manner. In this paper, we introduce a new framework named Med-Tuning to realize parameter-efficient tuning (PET) for medical volumetric segmentation task and an efficient plug-and-play module named Med-Adapter for task-specific feature extraction. With a small number of tuned parameters, our framework enhances the 2D baselines's precision on segmentation tasks, which are pre-trained on natural images. Extensive experiments on three benchmark datasets (CT and MRI modalities) show that our method achieves better results than previous PET methods on volumetric segmentation tasks. Compared to full FT, Med-Tuning reduces the fine-tuned model parameters by up to 4x, with even better segmentation performance. Our project webpage is at \url{https://rubics-xuan.github.io/Med-Tuning/}.

Main paper: 
 introduction: 
 Medical volumetric segmentation (MVS) task is to identify tumors and organ sub-regions in biomedical images, aiding accurate clinical diagnoses and treatment planning. It is crucial in medical research, due to the widespread use of 3D imaging like computed tomography (CT) and magnetic resonance imaging (MRI). In the last decades, a large number of deep neural network architectures have been proposed, including convolutional neural networks (CNNs) (e.g., (Milletari et al., 2016; C ¸içek et al., 2016; Isensee et al., 2021) ) and Transformer-based networks (e.g., (Cao et al., 2022; Hatamizadeh et al., 2022b,a; Zhou et al., 2023; Peiris et al., 2022) ). Recently, the "pre-training then fine-tuning" paradigm (Yosinski et al., 2014) has gained much popularity to enhance model performance in downstream tasks. As in (Cao et al., 2022) , the conventional full fine-tuning scheme updates all parameters of the pre-trained models. Yet, as models continuously improve in performance, particularly Transformer-based ones like (Cao et al., 2022; Hatamizadeh et al., 2022b,a) , their tuned parameter count escalates significantly. Thus, full fine-tuning involves a lot of tuned parameters and entails great training costs. To reduce tuned parameters, head-tuning (Head) was proposed (He et al., 2022) , focusing solely on optimizing the task-specific decoder, albeit resulting in decreased model performance. Meanwhile, recent studies (Jia et al., 2022; Chen et al., 2022; Pan et al., 2022; Sung et al., 2022; Yu et al., 2022; Zhang et al., 2023; Xu et al., 2023; Wu et al., 2023; Fischer et al., 2024) focus on parameter-efficient tuning (PET) to balance model performance and tuned parameters. In this paper, we aim to investigate how to adapt strong visual foundation models pre-trained on natural images to MVS tasks via PET. We initiate our analysis with some examples of widely available models that use image-level pretraining (e.g., classification task (Deng et al., 2009) , CLIP (Radford et al., 2021) , MOCO v3 (Chen et al., 2021) ) in natural image domain. Figure 1  
medical volumetric segmentation: 
 Achieving promising performance on MVS requires the incorporation of both spatial multiscale representations and volumetric correlations, as demonstrated by prior research (Hatamizadeh et al., 2022a) . Several U-Net inspired CNN-based models (Ronneberger et al., 2015; C ¸içek et al., 2016; Zhou et al., 2018; Isensee et al., 2021) concatenate multi-scale features from the encoder and up-sampled features, complementing the loss of spatial information caused by down-samplings. Cao et al. (Cao et al., 2022) use skip connections to effectively fuse low-level and high-level features in Transformers. Besides, various MVS methods capture volumetric correlations by 3D convolutions (C ¸içek et al., 2016; Milletari et al., 2016; Isensee et al., 2021) or self-attention mechanism among 3D volumes (Wang et al., 2021) .Recently, novel vision PET methods have emerged to balance accuracy and tuned parameter efficiency during fine-tuning, which can be categorized into three groups: (1) Prompt-based methods (Zhang et al., 2023; Fischer et al., 2024) . For instance, VPT (Jia et al., 2022) adds learnable prompt tokens to patch embeddings for downstream visual tasks. Pro-tuning (Nie et al., 2023) inserts multiple stage-wise prompt blocks into different stages of the backbone.(2) Adapter-based methods (Houlsby et al., 2019; Chen et al., 2022; Yang et al., 2023; Wu et al., 2023) . Adapter is a lightweight module inserted between the feed-forward layer and layer normalization in Transformer, which are tuned during fine-tuning while other layers stay frozen. ST-Adapter (Pan et al., 2022) introduces 3D depth-wise convolution (DWConv) (Ye et al., 2019) in Adapter modules to capture spatial-temporal features.(3) Other PET techniques. SAN (Xu et al., 2023) is a small and separate network that is trained via shortcut connections from backbone to reduce memory cost during fine-tuning.Recent studies (Wu et al., 2023; Chai et al., 2023) mainly focus on exploring the potential of the Segment Anything Model (SAM) for medical image analysis.Image analysis in Fourier domain is extensively used in various vision tasks (Ding et al., 2017; Lee et al., 2018; Li et al., 2020; Chi et al., 2020; Yang and Soatto, 2020; Rao et al., 2021) . Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT) leverage frequency information for global connectivity through parameter-free domain mapping on original images, resulting in an intrinsic global vision characteristic. According to the conclusion of (Oppenheim et al., 1979; Liu et al., 2023a) , the phase-only image or feature retains many of the semantics features of the original image. 
preliminaries: 
 Vanilla Adapter. Given an input feature X ∈ R N ×d , the vanilla Adapter can be represented as Equation (1) (Houlsby et al., 2019) , where W down and W up indicate the downprojection and up-projection layer, σ(•) is an activation function, + is a skip-connection.Adapter(X) = X + σ(XW down )W up , (1) Discrete Fourier Transform. Discrete Fourier Transform (DFT) serves as classical technique for computer vision applications (Rao et al., 2021) . Given a 3D input (volumetric data or feature) x[D, H, W ], DFT is defined as:EQUATIONwhere X is a complex matrix, R and I denote its real and imaginary part. In implementation, we use the accelerated versions of DFT and Inverse DFT (i.e., FFT and IFFT). Inadequate feature extraction will hinder performance even with the same robust decoder, as evidenced by the decline in results for Head compared to Full, detailed in Table 1 , Table 2 and Table 3 . Besides, not all segmentation decoders come with pre-trained weights, necessitating the fine-tuning of the entire decoder. Secondly, sole insertion in encoder part improves the flexibility of the whole framework. Our inserting strategy broadens the adaptability of Med-Tuning on visual foundation models while reducing tuned parameters.We propose a task-oriented and simple yet effective module for medical volumetric data, namely Med-Adapter. The motivation of Med-Adapter is to empower a 2D Transformer model pre-trained on natural images to gain the capability of volumetric feature modeling in a parameter-efficient manner. Here we consider three criteria when designing Med-Adapter: (1) MVS oriented : It's necessary to narrow the mentioned gaps in Figure 1 .(2) Light-weight: Structure with a low amount of parameters is crucial.(3) Plug-andplay: An easy-to-implement module is friendly to practical deployment. While retaining the bottleneck structure of the vanilla Adapter (Equation (1)), we introduce a few tailored designs into Med-Adapter based on above criteria, shown in Figure 2 (right).We introduce multiple branches tailored to capture fine-grained feature representations, coarse-grained global semantics and volumetric correlations among slices, which are vital for realizing accurate medical volumetric segmentation task.We employ parallel 3D convolutions with kernels of sizes k = 3 and k = 5 to extract spatial multi-scale features and 3D volumetric correlations between slices. The conventional 3D convolution operations are replaced with 3D DWConv in a parameter-efficient manner. Moreover, we use two cascaded 1 × k × k and k × 1 × 1 convolutions to replace the k×k×k convolution to pursue an extremely lightweight structure.FFT Global Branch (FGB) To achieve coarse-grained global semantic extraction in a parameter-efficient way, we substitute traditional large convolutional kernels and attention mechanisms, known for their memory and computation demands, with 3D FFT, IFFT and learnable complex matrices. These filter-like complex matrices are designed to model frequency features that contain global semantics in the whole frequency domain. Compared to vanilla 3D self-attention operation with O(n 3 ) complexity (n is the number of tokens), our FGB is a lightweight module. In detail, the computational complexity of FGB is O(nlog(n)), where FFT and IFFT are both with O(nlog(n)) complexity while that of Hadamard product or matrix addition is O(n). Then we merge the aforementioned branches by efficient 1 × 1 × 1 convolution. Therefore, given the intermediate embedded feature representations X ∈ [BD, C ′ , HW ], the Intra-FE module can be expressed as Equation ( 3)-( 5). In this way, the Intra-FE module is theoretically capable of modeling volumetric correlations among slices and incorporating abundant spatial multi-scale features for the downstream dense prediction task (i.e., MVS).EQUATIONEQUATIONX ′ = R volume (σ(XW down )), X ′ ∈ [B, C ′ , D, H, W ], C ′ = C/α (5)where M is the intra-stage enhanced feature representations. W F and b F are the learnable complex matrices, F is 3D FFT in Equation ( 2) and F -1 is 3D IFFT. ⊙ is the hadamard product, while R volume is a reshape operation to obtain cube-shape feature representations.To fully exploit the representations collected by Inter-FI modules in Med-Adapter at each stage, we further consider the feature interaction between different stages. The intra-stage enhanced feature representations M in this stage will be fused with M LastStage (i.e., the output of the Intra-FE module from the previous stage). Note that Inter-FI is only introduced at specific Med-Adapters at the end of each stage. Thus, Inter-FI is expressed as Equation ( 6). In this way, feature representations extracted by Intra-FE modules of Med-Adapters in shallow layers are gradually fed to adjacent higher layers, realizing inter-stage feature interaction by explicit enhancement and thus boosting model performance.Inter-FI = Cat(A(H, M LastStage )), M = Inter-FI, if F lag last M, if not F lag last (6)where A denotes convolution operations to align M and M LastStage in terms of spatial resolution and channel dimension, Cat refers the concatenation. F lag last is a bool parameter and F lag last = T rue when the current Med-Adapter is the inserted last one at this stage. In summary, our Med-Adapter can be formulated as Equation ( 7). R f latten denotes a symmetric operation that reshapes the feature back to the original shape of X. 
experimental setup: 
 Datasets and Evaluation Metrics. Our proposed Med-Tuning is evaluated on three benchmark datasets: (1) Kidney Tumor Segmentation 2019 (Heller et al., 2019 )(KiTS 2019), (2) Brain Tumor Segmentation 2019 (BraTS 2019) (Menze et al., 2014; Bakas et al., 2017 Bakas et al., , 2018)) , (3) Brain Tumor Segmentation 2020 (BraTS 2020) (Menze et al., 2014; Bakas et al., 2017 Bakas et al., , 2018)) , detailed in Appendix A.1. KiTS 2019 dataset comprises multi-phase 3D CTs depicting the kidneys and tumors. The ground truth contains 3 classes: background (label 0), kidney (label 1), and kidney tumor (label 2). The segmentation accuracy of KiTS 2019 is measured by kidney dice (label 1 and 2) and tumor dice (label 2), composite dice (the average of kidney and tumor dice). BraTS 2019 and BraTS 2020 datasets consist of 3D brain MRI scans with four modalities. The ground truth contains 4 classes: background (label 0), necrotic and non-enhancing tumor (label 1), peritumoral edema (label 2), and GD-enhancing tumor (label 4). The segmentation accuracy is measured by Dice score and the Hausdorff distance (95%) metrics for enhancing tumor region (ET, label 4), regions of tumor core (TC, labels 1 and 4), and whole tumor region (WT, labels 1,2 and 4). Implementation Details. Experiments utilizing PyTorch (Paszke et al., 2019) for implementation are conducted on NVIDIA GeForce RTX 3090 GPUs. The pre-trained vanilla ViT (Dosovitskiy et al., 2021) with UPerNet (Xiao et al., 2018) decoder (ViT+UPerNet) and Swin-UNet (Cao et al., 2022) based on pre-trained Swin Transformer(tiny) are our chosen baselines. More implementation details are available in Appendix A.2.We conduct experiments on three benchmark validation sets and compare our method with scratch (i.e., training with random initialization, without pre-training), full fine-tuning, head tuning and previous state-of-the-art PET approaches (i.e., VPT (Jia et al., 2022) , Adapter (Houlsby et al., 2019) , AdaptFormer (Chen et al., 2022) , Pro-tuning (Nie et al., 2023) , ST-Adapter (Pan et al., 2022) ). Qualitative results are shown in Appendix B. KiTS 2019. The performance comparisons with ViT+UPerNet and Swin-UNet as baseline are shown in Table 1 . Our proposed method boosts the performance of full fine-tuning considerably and achieves much higher Dice scores than previous PET methods, with much fewer tuned model parameters. In comparison with recently proposed PET methods (e.g., VPT, Pro-tuning and ST-Adapter), our Med-Tuning achieves better performance-efficiency trade-off on two baselines. Specifically, Med-Tuning improves model performance by a large margin (i.e., ↑ 4.20% Kidney Dice, ↑ 17.13% Tumor Dice, ↑ 10.67% Composite Dice on ViT+UPerNet and ↑ 1.01% Kidney Dice, ↑ 8.02% Tumor Dice, ↑ 4.52% Composite Dice on Swin-UNet) with only 17.70% and 27.58% of tuned parameters respectively in comparison with full fine-tuning. BraTS 2019. Performance comparisons on BraTS 2019 on two baselines are shown in Table 2 (left) and Table 3 (left) . Results show that our method attains the best trade-off between performance and efficiency, achieving comparable or even better results than previous methods. Compared to full fine-tuning, Med-Tuning achieves maximum improvements of 4.23% (ViT+UPerNet) and 1.28% (Swin-UNet) in Dice scores. Our Med-Tuning also achieves high parameter efficiency, tuning only 17.70% parameters of ViT and 27.58% of Swin-UNet, with inserted parameters being only 2.82% of ViT and 2.72% of Swin-UNet. BraTS 2020. Performance comparisons on BraTS 2020 are shown in Table 2 (right) and Table 3 (right). Compared to full fine-tuning, Med-Tuning achieves maximum improvements of 4.07% (ViT+UPerNet) and 1.64% (Swin-UNet) in Dice scores with very few tuned parameters, surpassing most of PET methods. Compared to ST-Adapter, our tuned parameters are fewer yet yield a more substantial overall performance improvement. Moreover, Med-Tuning took about 1.34 (ViT+UPerNet) and 1.68 (Swin-UNet) hours for fine-tuning, 0.76 (ViT+UPerNet) and 0.46 (Swin-UNet) minutes per sample for inference. Extensive ablation experiments are conducted based on five-fold cross-validation. For more ablation experiments please refer to Appendix C.Inserted Position of Med-Adapter. We conduct experiments on BraTS 2019 training set to assess the segmentation performance by inserting Med-Adapter at various stages of Swin-UNet encoder. Given that Swin-UNet encoder has four continuous stages (n = 0, 1, 2, 3). According to Table 4 , Inserting Med-Adapter in the initial stages resulted in degraded performance, with none surpassing our best default setting (gray background). This may be attributed to the greater contribution of features learned in later encoder stages when transferring pre-trained weights to the MVS task. Generalization Capability on Other Pre-trained Weights. To explore the potential of our Med-Tuning, we investigate the effect of diverse encoder pre-trained weights (e.g., multi-modal based (CLIP (Radford et al., 2021) ), self-supervised based (MAE (He et al., 2022) , MoCo v3 (Chen et al., 2021) ) and SAM (Kirillov et al., 2023) ) on BraTS 2019 training set with ViT-B/16. As presented in Table 5 , given different pre-trained weights, our easy-to-integrate framework boosts the performance consistently with much fewer tuned parameters, suggesting the effectiveness and robustness of our Med-Tuning framework. Generalization Capability on 3D Baseline and Medical Pre-trained Weight. To demonstrate the generalization capability of our approach, we select Swin UNETR (Tang et al., 2022) pre-trained on medical datasets as a supplementary 3D baseline and experiment on part of the Medical Segmentation Decathlon (MSD) (Antonelli et al., 2022) dataset. For implementation details please refer to Appendix A.3. Experimental results in Table 6 show that our method still outperforms full fine-tuning in Memory, Time and Dice score.  
conclusion: 
 In this work, we present a new PET framework named Med-Tuning with strong generalization capabilities for the practical application of MVS. Taking advantage of both spatial relationship modeling (coarse/fine-grained) and volumetric correlations, our framework achieves better volumetric segmentation accuracy on 2D baselines pre-trained on relatively easily acquired natural images. To some extent, Med-Tuning could consistently and sustainably boost the segmentation performance of pre-trained models on MVS tasks, keeping pace with the rapid development of foundation models in computer vision field. (2) random mirror flipping across the axial, coronal and sagittal planes by a probability of 0.5; (3) random intensity shift between [-0.1, 0.1] and scale between [0.9, 1.1]. L2 Norm is also applied for regularization with a weight decay rate of 10 -5 . As for the KiTS 2019 dataset, the employed data augmentations follow as the prior work (Isensee et al., 2021) . We conduct ablation experiments to investigate the generalization capability of our Med-Tuning on the 3D baseline and pre-trained weight on the medical dataset. The Medical Segmentation Decathlon (MSD) (Antonelli et al., 2022) dataset includes 10 segmentation tasks covering various organs and image modalities. These tasks are intentionally diverse, presenting challenges like limited training data, class imbalances, multi-modality data, and small objects. In the main text, we have validated our approach on two MRI and one CT benchmark dataset. For ablation experiments, we selected one MRI and two CT datasets (i.e., Task02 Heart (MRI), Task06 Lung (CT), and Task09 Spleen (CT)) from the MSD dataset. Dataset pre-processing followed the protocol outlined in Swin UNETR (Tang et al., 2022) . In Table 6 , Memory(GB) represents memory usage during the fine-tuning, Time(h) denotes the fine-tuning time, and Dice AVG signifies the average of multi-class Dice scores for the corresponding segmentation task.A.4. The position of the inserted parameters.Regarding the insertion position for the parameters, for SwinUnet-Tiny and SwinUNETR, we have incorporated the Med-Adapter exclusively within each transformer layer of their encoder. This results in a total of 8 Med-Adapters, calculated from 4 × 2 (number of stages × number of layers in each stage). Within each stage, the second Med-Adapter is designated for Inter-FI. In the case of ViT+UPerNet, a Med-Adapter is inserted following every layer, amounting to a total of 12 (number of layers in ViT-B/16) Med-Adapters. Specifically, the Med-Adapters placed after the 2nd, 5th, 8th, and 11th layers are used for Inter-FI, maintaining a division of the ViT encoder into 4 stages, similar to the Swin Transformer setup. The relative positioning between Med-Adapters and Transformer blocks can be referenced in Figure 2 of our manuscript. Through our experiments, we have determined that the insertion position illustrated in Figure 2 of the manuscript represent the optimal configuration, as currently established.A.5. The scale of bottleneck features of Med-Adapter.As indicated in Table 9 of our manuscript, the default Reduction Ratio is set to 6. Consequently, for all baselines, the scale of the bottleneck features of the adapter is represented by L/6, L is the base scale of features in each stage (i.e., the scale of input features of Med-Adapter). Specifically, the scale of bottleneck features of 8 Med-Adapter in SwinUnet-Tiny or SwinUNETR are [16, 16, 32, 32, 64, 64, 128, 128] . The scale of bottleneck features of 12 Med-Adapter in ViT+UPerNet are both 128. 
 
 Appendix: Comparison with full fine-tuning, head tuning and previous PET methods in terms of the trade-off between the number of tuned parameters and segmentation accuracy is shown in Figure 3 . The experiments were conducted using ViT+UPerNet as the baseline on the BraTS 2019 dataset. The horizontal axis represents the parameters involved in model training during the fine-tuning stage, while the vertical axis denotes the mean Dice scores for ET, WT, and TC. Our method achieves much better segmentation performance than full fine-tuning and previous state-of-the-art PET methods with much less tuned parameters.Qualitative results of BraTS 2019 datasets are shown in Figure 4 , with the comparison with full fine-tuning, ST-Adapter and VPT. As the labels for the validation set are not available, five-fold cross-validation is conducted on the training set for visualization. Our method recognizes brain tumors in enhancing and non-enhancing regions more accurately and reduces missed or false identification of the peritumoral edema in general. As the labels for the validation set are not available, five-fold cross-validation is conducted on the training set for visualization. The qualitative results of KiTS 2019 datasets, depicted in Figure 5 , highlight the superior performance of our method in organ and tumor segmentation compared to full fine-tuning, ST-Adapter, and VPT. Our approach demonstrates enhanced accuracy in segmenting organs and tumor types, producing finer-grained segmentation masks for corresponding tumors. C.4. Inter-FI Module Design.After investigating the effect of the intra-stage feature enhancement, we further verify the effectiveness of the inter-stage feature interaction, as shown in Table 12 . Compared with the intra-only structure (i.e., without the feature connectivity between adjacent Med-Adapters), the model with inter-stage achieves a considerable performance gain with only 0.319M extra parameters for feature alignment among adjacent stages, showing the effectiveness of our inter-stage interaction. Unlike concatenation which maintains the feature representations of different stages as much as possible, direct addition or taking the maximum value (at each pixel) of neighboring feature maps with diverse semantic levels would unintentionally degrade the original feature representation, resulting in a sharp decrease in segmentation performance.C.5. Decoder Design.Here we explore the effect of different decoder designs in our architecture. Although the backbone is frozen and only the inserted Med-Adapters as well as the decoder are updated during fine-tuning, the essentially tuned model parameters introduced by the segmentation decoder can not be reckoned as negligible. In other words, to pursue an extremely PET framework, the design of the employed decoder should be sufficiently lightweight with strictly controlled model parameters. Thus, various segmentation decoders with greatly varied model complexity are introduced respectively for a thorough analysis. As shown in Table 13 , ViT-B/16 with the SETR-MLA decoder reaches the best trade-off between segmentation accuracy and tuned parameter costs, benefiting from the effective multi-scale At last, we also explore the data efficiency property of our method by examining performance across various training data ratios, particularly in low-data settings. Table 14 shows the quantitative comparison with different numbers of training samples. Our Med-Tuning can already achieve comparable performance to full fine-tuning using only 25% training data. As the scale of training data increases, our method consistently improves the segmentation accuracy, with reduced training time and memory cost compared with full fine-tuning. C.7. Other Weight Pre-trained on Medical Image Datasets.Med-Tuning is not solely focused on pushing SOTA. Instead, it allows us to capitalize on the extensive progress made in natural image processing. This perspective underscores our belief in the potential and value of integrating advancements from one domain to enhance the capabilities and applications in another.Indeed, as highlighted in recent literature (Liu et al., 2023b; Silva-Rodríguez et al., 2023; Ulrich et al., 2023) , there have been significant advancements in the field of medical image pre-trained models. Nevertheless, due to the considerable constraints of time, monetary resources, and clinical applicability faced by many researchers working on medical image pre-training, the pace of updates and the scale of medical image pre-training efforts still trail behind those in the natural image domain. Additionally, the use of many open-source codes in the medical imaging field presents a high threshold. Therefore, the vast array of convenient and accessible large-scale pre-trained weights from the natural image domain have become our primary choice.Based on the above choices, we hypothesize that: If Med-Tuning can tackle the more challenging task of a large domain shift from features pre-trained on natural 2D images to CT/MRI volumes, then it is also capable of addressing the comparatively easier task of domain shift from features pre-trained on medical images to CT/MRI volumes. The experimental results in our manuscript have demonstrated the feasibility of a broader transfer process, thereby validating the effectiveness of our proposed approach in achieving the former scenario.Regarding the latter scenario, we conducted experiments using the same baselines and pre-trained weights as those in (Liu et al., 2023b; Silva-Rodríguez et al., 2023) , following our default training setting. The comprehensive results of all experiments are depicted in Table 15 .Table 15 : The comparison between original SwinUNETR, Universal Model and our proposed Med-Tuning. The performance is evaluated by average Dice scores. "W1" signifies the use of the model and pre-training weights from (Tang et al., 2022) , while "W2" references the model and pre-training weights from (Liu et al., 2023b) . "SCR" denotes the model is trained from scratch and "FULL" denotes the full fine-tuning mothod. The first two columns of scores were directly copied from (Liu et al., 2023b) . The last four columns of scores were obtained through training using our framework. From our results, Med-Tuning proves to be capable of consistently improving the precision in medical volumetric segmentation tasks by using medical pre-trained weights, requiring only a small number of training parameters for this enhancement. Besides, the trends observed in our experimental results suggest that our proposed approach can keep pace with the development of visual models pre-trained in medical domain, aligning with the conclusions drawn at the end of our manuscript. Finally, we would like to add that as demonstrated in our results shown in Table 10 , using the W2 weights improved the Dice score by 0.6 over the W1 weights. Hence, we also look forward to the widespread development of large-scale pre-trained models like (Liu et al., 2023b) in medical domain and are excited about the potential to further enhance their performance using our Med-Tuning.Under default training settings, the training time of each method are listed in Table 16 . The results indicate that the introduction of few new training parameters inevitably results in a slight increase in training. However, we achieves a commendable balance between training time cost and the tuning of parameters. 



Reviews: review: paper_summary: paper_summary: This paper describes a parameter-efficient tuning (PET) method to adapt foundation models to the task of 3D medical image segmentation. They present three different contributions: 1) The Med-Tuning Framework, 2) The Med-Adapter Module, and 3) Extensive experiments on different modalities of medical imaging (MRI, CT) and ablation studies. In this framework, authors show the potential of harnessing pre-training done on foundation models to enhance segmentation performances and reduce training costs by only training parts of the model.
main_review: main_review: The authors have addressed all my questions and interrogations. My final rating is therefore increased.
Indeed, some improvements seem to not be significant but interesting discussions can rise from this work.
strength_weakness: strength_weakness: Although the performances shown are superior to previous methods, I would have liked to see confidence intervals/standard errors of the performances as the results between the methods are very close. If there is an overlap of the results, one could not confidently conclude that such method performs best. 
Another point that was a bit surprising to me was the lack of difference in performance shown in the ablation study. If I understood correctly the first line of Table 4 represents no presence of Med-Adapter, and the last line is your proposed method. Although the average values of your performances are superior, I wish the authors presented standard errors to ensure no to little overlap of the confidence intervals to effectively conclude on an improvement.
One last comment I would have is on the concept itself of PET. I would be curious to see how your method fares compared to training a nnUnet as an even simpler baseline. To my knowledge, nnUnet has always been shown to outperform other methods while being much lighter. Fine-tuning is an interesting process when very little data is present for training and testing but in the datasets you have here, such verification would be possible and empower your claim to use PET methods over retraining a model (albeit much lighter) from the ground up.
questions: questions: Cf weaknesses and detailed comments
paper_summary: paper_summary: This manuscript present the MedTuning framework that proposes a way to fine-tune foundational models pre-trained on 2D images for medical imaging segmentation using a Fourier transform based adapter inserted in the decoding part of the network at a lesser cost in terms of memory and tuned parameters
Experiments are run in comparison to other fine-tuning methods over the KiTS and BRATS datasets using two different baseline architectures, namely SwinUNet and ViT + UPerNet.
main_review: main_review: The authors have interestingly addressed most of the comments and have put some effort in clarifying their statements and their claims. Given the possible interest of the community in the proposed solution, I would be happy to move my score to a borderline accept
strength_weakness: strength_weakness: - The overall clarity of the paper is quite poor and it becomes difficult to follow the rationale behind the methods.
- The link between the equations and the text is sometimes unclear. For instance, when reading the description of the Adapter, a natural translation would be $Adapter(X) = W_{up}(\sigma(W_{down}(X)))$. It is quite unclear where the addition term X+ comes from from the textual description
- There is no indication of results variability and no adequate statistical testing which may lead to overstatements of the results.
questions: questions: Please re-ascertain the strengths of the results given adequate statistical analysis of the performance comparisons and temper the conclusions accordingly.
paper_summary: paper_summary: This paper proposes to adapt backbones pre-trained on natural image classification, segmentation, or vision-language pre-training, for volumetric medical image segmentation. To do so, their Med-Tuning method includes special modules for transferring 2d architectures to 3d data, using adapters with 3D convolutions. Also, Med-Tuning includes inter-level feature concatenation, and FFT transforms. Med-Tuning is validated across several tasks using CT and MRI volumes, using different backbones.
main_review: main_review: I thank the authors for their detailed responses. However, after carefully reading the comments of other reviewers and the authors' responses, I believe that some important concerns have not been adequately addressed, which motivates me to lower my score. I include below the detailed weaknesses that motivate my decision.

1. **Experimental Setting and Baselines**. Focusing on the large-data regime, i.e. several hundred volumes for training, as stated by Reviewer ufgr, dataset-specific models (instead of adapted ones) might be an appealing solution, using dataset-specific architectures such as nnUnet, in particular its 3D version. These solutions are not considered in the manuscript. In addition, Scratch or Full training baselines (Table 1, 2, and 3) only receive 2D information, thus being a weakly implemented baseline compared to the 3D-based Adapter.


2. Med-Tuning framework introduces several **hyperparameters and modules**, evaluated in several ablation experiments (Table 4, Table 9, Table 10, Table 11, Table 12). Each of such decisions might vary (1%-2%) the obtained results. There are some tasks (e.g. BraTS) for which performance improvements are minor, which might question the actual performance gains to previous adapters if going through such empirical refinement.


3. **Med-Adapter configuration**. As questioned by Reviewer XA6b, the proposed adapter fails to integrate a motivated link for the addition of several components (e.g. FGB and Inter-FI), which are unproperly linked to previous literature adapters. In addition, some branches, such as inter-FI, introduce minor performance gains (Table 12).


4. **Comparison with 3D medical pre-trained foundation models**. The authors include in Table 6 a SwinUNETR model pre-trained uniquely on 1 dataset. Initial rebuttal experiments (Table 10 at Reviewer H82q discussion) suggest already an improvement when using large-scale pre-trained medical volumetric models with the proposed Med-Tuning adapter compared to Table 6 experiments. It is worth mentioning that such experiments involve fine-tuning the pre-trained decoder, which might distort rich pre-trained specialized features [e]. Such domain-specific models might require less tuning than natural-image pre-trained models, and thus, I do not agree that “by conducting comparisons within each pre-trained domain separately, we can adequately demonstrate the superiority of Med-Tuning”. As pointed out in my initial review, there exists a growing appearance of medical pre-trained volumetric models [a,b,c,d], some of them exploring parameter-efficient alternatives that benefit from the pre-trained decoder (e.g. Linear Probe then Fine-Tune in [c] or spatial adapters in [b]). Thus, the transferability of natural-image 2d pre-trained models for transferability to 3D medical image segmentation is not properly justified.

[a] CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection, ICCV, 2023.

[b] Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentation, MICCAI MedAGI Workshop, 2023.

[c] MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation, MICCAI, 2023.

[d] UniSeg: A Prompt-driven Universal Segmentation Model as well as A Strong Representation Learning, MICCAI, 2023.

[e] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, ICLR, 2022.
strength_weakness: strength_weakness: - The core idea of parameter-efficient fine-tuning implies re-using rich pre-trained features for segmentation and adapting them to downstream tasks by modifying a small set of new or existing parameters. Under this scope, using features pre-trained on natural images, and 2D data, proposes a large domain shift to the target domain: CT/MRI volumes. In addition, some dedicated designs, such as the proposed multi-scale local branch, are necessary to adapt the 2D model to 3D data. Thus, adapting such models, instead of recently released foundation models for 3D data (see [a,b,c]), might be unnecessary and suboptimal.
- Following the previous comment, there is one experiment using Swin UNETR, but this seems pre-trained only on 1 dataset, in contrast to [a,b,c]. Also, there is no direct comparison with natural image pre-training.
- Although the number of parameters is indicated for all PEFT methods in Table 1, this is not the case for training times. For some settings, training times might be larger even when using a small set of parameters, due to convergence speeds.
- Following the previous comment, some training details are missing: are the authors using a validation set? Are authors doing early stopping? Some methods might converge faster than others.
- Implementation details for baselines. There are some missing implementation details, mostly for the baseline methods. This might question if all baseline methods were properly adjusted (i.e. learning rate, the position of the inserted  parameters, scale of bottleneck features of the adapter, etc.).

[a] CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection, ICCV, 2023.

[b] Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentation, MICCAI MedAGI Workshop, 2023.

[c] MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation, MICCAI, 2023.
questions: questions: Please, see Weaknesses.

Comments: title: Sincerely appreciation for novelty, contribution, and the positive feedback
Sincerely appreciation for your recognition of the novelty, contributions, extensive experiments and clear identification of the current gaps
Sincerely appreciation for your recognition of our extensive literature review, experiments, and good writing
comment: Many thanks for your kind comments and great interests on our work. Please let me clearly clarify this last question in the followings.

**1. Decoder training**

As indicated in the methodological section 3.2 and Figure 2 of our paper, Med-Tuning incorporates a pre-trained backbone along with a segmentation decoder that does not inherently require pre-trained weights for its operation.

To provide further detail:
- On one hand, in cases where a backbone such as ViT consists of only an encoder, it is necessary to integrate an effective decoder to address our segmentation tasks. In such scenarios, training the decoder from scratch becomes inevitable.
- On the other hand, when a backbone already has a pre-trained encoder-decoder architecture, like SwinUnet and SwinUNETR, we directly utilize its weights to initialize the parameters for both the encoder and decoder. In these instances, the decoder can be trained using a full fine-tuning approach. Specifically for SwinUnet, we adhere to the method detailed in its original paper for symmetrically initializing the decoder with the pre-trained weights of Swin Transformer.

**2. Med-Adapter implementation details**

In reference to the nuanced implementation details of the Med-Adapter, we wish to provide clarification regarding the adjustments made to accommodate the SwinUNETR architecture, which serves as a 3D baseline.
Specifically, the downsample and upsample linear layers have been adapted to utilize downsample and upsample 3D convolutions with a kernel size of 1. 
Moreover, the reshape operations traditionally require before and after the Depth-Wise Convolution, denoted as $R_{volume}$ and $R_{flatten}$ in equations 5 and 7 of our manuscript, have been found unnecessary for SwinUNETR and thus have been omitted. 

These modifications constitute the only deviations from our Med-Adapter’s standard implementation. 

We hope these detailed explanations above illuminate the bespoke adaptations implemented for the SwinUNETR architecture in our study, giving us a precious opportunity to strive for your raised final rating.
Thank you for the detailed responses to my comments. However, I have one last question that I would kindly ask the authors to clarify:

Regarding the experiments using SwinUNETR pre-trained on medical datasets (Table 6 of the paper and new experiments in rebuttal Table 10 using UniversalModel). How is Med-Tuning implemented in this case? SwinUNETR already has a pre-trained encoder-decoder architecture for volumetric data. For example, the methodological description of Med-Tuning (Section 3.2 and Figure 2) involves training a decoder from scratch.
> [**Q5**]. Implementation details for baselines. There are some missing implementation details, mostly for the baseline methods. This might question if all baseline methods were properly adjusted (i.e. learning rate, the position of the inserted parameters, scale of bottleneck features of the adapter, etc.). & Nevertheless, I have some concerns about the efficiency of ... and experimental designs for baseline methods.

[**A5**]. 

**(1) Implementation details for 3 baselines.** Below, we list some of the config used for the SwinUnet and ViT+UPerNet baselines in our manuscript. The configs of SwinUNETR is consistent with [1].
```Python
# SwinUnet
_C.MODEL.SWIN.PATCH_SIZE = 4
_C.MODEL.SWIN.EMBED_DIM = 96
_C.MODEL.SWIN.DEPTHS = [2, 2, 2, 2]
_C.MODEL.SWIN.DEPTHS_DECODER = [2, 2, 2, 1]
_C.MODEL.SWIN.NUM_HEADS = [3, 6, 12, 24]
_C.MODEL.SWIN.WINDOW_SIZE = 4
_C.MODEL.SWIN.MLP_RATIO = 4.
_C.MODEL.SWIN.PRE_CKPT='../../ckpt/swin_tiny_patch4_window7_224.pth' # supervised pre-training

# ViT+UPerNet
_C.MODEL.VIT.PATCH_SIZE=16
_C.MODEL.VIT.EMBED_DIM=768
_C.MODEL.VIT.DEPTH=12
_C.MODEL.VIT.NUM_HEADS=12
_C.MODEL.VIT.PRE_CKPT_SUP='../../ckpt/ViT-B_16.npz' # supervised pre-training
```

**(2) Learning rate:** We have identified the optimal hyperparameter settings for different baselines through extensive experiments. For the experiments on the BraTS 2019 & BraTS 2020 datasets, the learning rate is set at 0.002 for SwinUnet and 0.002 for ViT+UPerNet. For the experiments on the KiTS 2019 dataset, the learning rate is 0.002 for SwinUnet and 0.004 for ViT+UPerNet. Additional implementation details, such as training epochs and warm-up periods, are provided in Table 12.

**Table 12: Implementation details.**
| Dataset | Baseline | Encoder Backbone | Learning Rate| Training Epochs| Warm-up Epochs|
|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|
|BraTS 2019 & BraTS 2020|SwinUnet|Swin-Tiny|0.002|250|60|
|BraTS 2019 & BraTS 2020|ViT+UPerNet|ViT-B/16|0.002|250|25|
|KiTS 2019|SwinUnet|Swin-Tiny|0.002|500|20|
|KiTS 2019|ViT+UPerNet|ViT-B/16|0.004|500|20|
|MSD Task02 Heart|SwinUNETR|Swin-Tiny|0.005|400|40|
|MSD Task06 Lung|SwinUNETR|Swin-Tiny|0.005|400|50|
|MSD Task09 Spleen|SwinUNETR|Swin-Tiny|0.005|400|40|

**(3) The position of the inserted parameters:** Regarding the insertion position for the parameters, for SwinUnet-Tiny and SwinUNETR, we have incorporated the Med-Adapter exclusively within each transformer layer of their encoder. This results in a total of 8 Med-Adapters, calculated from $4\text{ (number of stages)}×2\text{ (number of layers in each stage)}$. Within each stage, the second Med-Adapter is designated for Inter-FI. In the case of ViT+UPerNet, a Med-Adapter is inserted following every layer, amounting to a total of $12\text{ (number of layers in ViT-B/16)}$ Med-Adapters. Specifically, the Med-Adapters placed after the $2nd, 5th, 8th,$ and $11th$ layers are used for Inter-FI, maintaining a division of the ViT encoder into 4 stages, similar to the Swin Transformer setup. The relative positioning between Med-Adapters and Transformer blocks can be referenced in Figure 2 of our manuscript. Through our experiments, we have determined that the insertion position illustrated in Figure 2 of the manuscript represent the optimal configuration, as currently established.

**(4) The scale of bottleneck features of Med-Adapter:** As indicated in Table 9 of our manuscript, the default Reduction Ratio is set to 6. Consequently, for all baselines, the scale of the bottleneck features of the adapter is represented by $L/6$, $L$ is the base scale of features in each stage (i.e., the scale of input features of Med-Adapter). Specifically, the scale of bottleneck features of 8 Med-Adapter in SwinUnet-Tiny or SwinUNETR are $[16, 16, 32, 32, 64, 64, 128, 128]$. The scale of bottleneck features of 12 Med-Adapter in ViT+UPerNet are both $128$.

[1] Tang, Y., Yang, D., Li, W., Roth, H.R., Landman, B., Xu, D., Nath, V., Hatamizadeh, A.: Self-supervised pre-training of swin transformers for 3D medical image analysis. In: CVPR (2021)
> [**Q3**]. Although the number of parameters is indicated for all PEFT methods in Table 1, this is not the case for training times. For some settings, training times might be larger even when using a small set of parameters, due to convergence speeds.

[**A3**]. Under default training settings, the training time of each method are listed in Table 11. The results indicate that the introduction of few new training parameters inevitably results in a slight increase in training. However, we achieves a commendable balance between training time cost and the tuning of parameters.

**Table 11: Comparisons of training time (hours) on BraTS2019 with SwinUnet and ViT+UPerNet backbone.**
| Method | ViT+UPerNet | SwinUnet |
|:------|:-------:|:-------:|
| Scratch | 1.74 | 	1.26 
| Full | 1.73 | 1.26
| Head | 1.28 | 1.02
| VPT-Shallow | 1.09 | 0.98
| VPT-Deep | 1.18 | 1.01
| Adapter | 1.77 | 1.30
| AdaptFormer | 1.44 | 1.18
| Pro-tuning | 1.84 | 1.47
| ST-Adapter | 1.79 | 1.55
| Ours | 1.88 | 1.51


> [**Q4**]. Following the previous comment, some training details are missing: are the authors using a validation set? Are authors doing early stopping? Some methods might converge faster than others. 

[**A4**]. Due to the relatively small size of medical imaging datasets, it is vital to utilize the entire official training data. Therefore, we trained our models in the following manner:

Firstly, we initially employed 3-fold cross-validation, dividing the entire official training set into three parts. Each part was alternately used as the validation set, with the remaining two parts serving as the training set, thereby creating three sets of train-validation pairs. 

With each change in hyperparameters, we conducted training and validation across these three train-validation pairs, ultimately calculating their average performance as the representation of effectiveness on the train-validation pairs. Through iterative adjustments of the hyperparameters, we identified a set of hyperparameters that exhibited optimal average performance across the train-validation pairs. 

Subsequently, employing this optimal set of hyperparameters, we utilized all data from the official training set for training, followed by testing on the official test set and submitting these results to the official website for scoring. This experiment was replicated three times, and the mean of these repetitions was used to determine the scores presented in Tables 1, 2, and 3 of our manuscript.
**Table 10: The 5-fold cross-validation performance on MSD. These are the comparison between original SwinUNETR, Universal Model and our proposed Med-Tuning. The performance is evaluated by average Dice scores. "W1" signifies the use of the model and pre-training weights from [d], while "W2" references the model and pre-training weights from [a]. "SCR" denotes the model is trained from scratch and "FULL" denotes the full fine-tuning mothod. The first two columns of scores were directly copied from [a]. The last four columns of scores were obtained through training using our framework.**
| Dataset | SwinUNETR (SCR) | Universal Model (FULL) | Ours (SCR) | Ours (FULL, W1) | Ours (Med-Tuning, W1) | Ours (FULL, W2) | Ours (Med-Tuning, W2) |
|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|
|Task06 Heart| 68.90 |67.15 | 65.82 | 67.69 | 78.09 | 68.37 | 78.53 |
|Task09 Heart| 95.80 |96.71 | 95.76 | 96.52 | 97.06 | 96.35 | 97.60 |

From our results, Med-Tuning proves to be capable of consistently improving the precision in medical volumetric segmentation tasks by using medical pre-trained weights, requiring only a small number of training parameters for this enhancement. Besides, the trends observed in our experimental results suggest that our proposed approach can keep pace with the development of visual models pre-trained in medical domain, aligning with the conclusions drawn at the end of our manuscript.

Finally, we would like to add that as demonstrated in our results shown in Table 10, using the W2 weights improved the Dice score by 0.6 over the W1 weights. Hence, we also look forward to the widespread development of large-scale pre-trained models like [a] in medical domain and are excited about the potential to further enhance their performance using our Med-Tuning.

[a] CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection, ICCV, 2023.

[b] Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentation, MICCAI MedAGI Workshop, 2023.

[c] MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation, MICCAI, 2023.

[d] Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis. CVPR, 2021.


> [**Q2**]. Also, there is no direct comparison with natural image pre-training.

[**A2**]. By conducting comparisons within each pre-trained domain separately, we can adequately demonstrate the superiority of Med-Tuning. Thus, it is not essential to use the same baseline for a comparative analysis of Med-Tuning across both pre-trained domains simultaneously. Therefore, we believe that the existing experiments already provide convincing evidence to support our claims. Besides, it appears that among the current open-source pre-trained models, there are few that possess both natural image pre-training and medical image pre-training weight. We sincerely hope for your understanding and welcome any further discussions or suggestions regarding our work.
Thanks very much for acknowledging the novelty and contributions of our paper, as well as the provided positive feedback "The methodological description of the paper is sound, and the authors provide many experiments and adequate ablation experiments." on our extensive evaluations and robustness. 

We have addressed all your questions in detail in the Response section below. It is our sincere hope that our thorough explanations, coupled with additional experiments conducted to the best of our abilities, will contribute to an improved evaluation score of our work.


**Our Responses to Paper Weaknesses:**

> [**Q1**]. The core idea of parameter-efficient fine-tuning implies re-using rich pre-trained features for segmentation and adapting them to downstream tasks by modifying a small set of new or existing parameters. Under this scope, using features pre-trained on natural images, and 2D data, proposes a large domain shift to the target domain: CT/MRI volumes. In addition, some dedicated designs, such as the proposed multi-scale local branch, are necessary to adapt the 2D model to 3D data. Thus, adapting such models, instead of recently released foundation models for 3D data (see [a,b,c]), might be unnecessary and suboptimal.Following the previous comment, there is one experiment using Swin UNETR, but this seems pre-trained only on 1 dataset, in contrast to [a,b,c]. & Nevertheless, I have some concerns about the efficiency of adapting natural image pre-trained models.

[**A1**]. 

Thanks for this feedback. We appreciate the opportunity to clarify this point and remain open to further discussion that might arise regarding our work.

**Firstly, we would like to assert that while our proposed method may be considered 'suboptimal,' it is certainly not 'unnecessary.'**

Med-Tuning is not solely focused on pushing SOTA. Instead, it allows us to capitalize on the extensive progress made in natural image processing. This perspective underscores our belief in the potential and value of integrating advancements from one domain to enhance the capabilities and applications in another. 

Indeed, as highlighted in recent literature [a][b][c], there have been significant advancements in the field of medical image pre-trained models. Nevertheless, due to the considerable constraints of time, monetary resources, and clinical applicability faced by many researchers working on medical image pre-training, the pace of updates and the scale of medical image pre-training efforts still trail behind those in the natural image domain. Additionally, the use of many open-source codes in the medical imaging field presents a high threshold. Therefore, the vast array of convenient and accessible large-scale pre-trained weights from the natural image domain have become our primary choice.

Based on the above choices, we hypothesize that: If Med-Tuning can tackle the more challenging task of *a large domain shift from features pre-trained on natural 2D images to CT/MRI volumes*, then it is also capable of addressing the comparatively easier task of *domain shift from features pre-trained on medical images to CT/MRI volumes*. The experimental results in our manuscript have demonstrated the feasibility of a broader transfer process, thereby validating the effectiveness of our proposed approach in achieving the former scenario. 

Regarding the latter scenario, we conducted experiments using the same baselines and pre-trained weights as those in [a][b], following our default training setting. The comprehensive results of all experiments are depicted in Table 10.
> [**Q5**]. The idea is quite interesting with comprehensive experiments but the overall clarity is poor and the conclusions may overstate the results. Statistical analysis is fully lacking and there is no clear link between the different pieces of the framework in terms of justification and/or explanation.

[**A5**]. Here, we provide a summary of our detailed responses above:
1. We enhanced the overall clarity of the paper and improved certain inappropriate expressions.
2. We supplement some representative raw results accompanied by their standard deviations for statistical analysis, which demonstrate that there is no overstatement of our results.

We hope the responses provided above address any concerns or questions you may have regarding our paper.
> [**Q3**]. There is no indication of results variability and no adequate statistical testing which may lead to overstatements of the results. & Please re-ascertain the strengths of the results given adequate statistical analysis of the performance comparisons and temper the conclusions accordingly.

[**A3**]. Per your advice, we supplement some representative raw results accompanied by their standard deviations for statistical analysis, which are listed below in Table 6-9 for your convenience to check. The results indicate that, despite **minor overlaps in certain cases**, our method is **generally comparable to or surpasses** the results of full fine-tuning and most of the previous PET methods.

**Table 6: Performance comparison on KiTS 2019 with SwinUnet backbone.**
| Method | Kidney Dice | Tumor Dice | 
|:------|:-------:|:-------:|
| Scratch | 94.33±0.06 | 61.10±0.40 |	 
| Full | 94.68±0.07 | 62.13±0.97 |
| Head | 91.95±0.03 | 53.93±0.35 |
| VPT-Shallow | 91.72±0.09 | 54.86±0.71 |
| VPT-Deep | 91.53±0.12 | 53.41±0.76 |
| Adapter | 93.02±0.04 | 57.15±0.62 |
| AdaptFormer | 93.74±0.03 | 59.79±0.98 |
| Pro-tuning | 90.34±0.08 | 51.19±0.46 |
| ST-Adapter | 92.97±0.07 | 57.33±0.53 |
| Ours | 95.69±0.05 | 70.14±0.54 |

**Table 7: Performance comparison on BraTS2019 with SwinUnet backbone.**
| Method | Dice(ET) | Dice(WT) | Dice(TC) | Hausdorff(ET)| Hausdorff(WT)| Hausdorff(TC)|
|:------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|
Scratch|78.38±0.29|88.59±0.12|76.46±0.45|6.055±0.202|10.651±0.531|9.176±0.301|
Full|78.26±0.25|89.56±0.07|79.16±0.63|4.327±0.598|6.149±0.179|6.704±0.120|
Head|78.07±0.07|88.68±0.09|77.26±0.33|5.021±0.374|6.697±0.082|7.091±0.392|
VPT-Shallow|77.16±0.18|88.30±0.15|76.77±0.27|5.421±0.108|6.154±0.044|7.345±0.053|
VPT-Deep|77.02±0.22|88.65±0.07|76.91±0.32|5.297±0.220|7.088±0.157|7.940±0.132|
Adapter|77.98±0.25|89.22±0.19|78.02±0.46|5.298±0.061|6.622±0.094|8.490±0.359|
AdaptFormer|77.69±0.13|88.61±0.14|76.83±0.68|4.909±0.802|6.290±0.058|7.885±0.287|
Pro-tuning|78.58±0.14|89.33±0.04|78.79±0.36|5.273±0.040|6.410±0.323|8.236±0.376|
ST-Adapter|78.40±0.19|89.54±0.02|77.44±0.11|4.751±0.068|6.013±0.037|7.405±0.317|
Ours|78.51±0.16|89.68±0.03|80.44±0.12|4.003±0.062|5.517±0.043|5.756±0.304|

**Table 8: Performance comparison on BraTS2019 with ViT+UPerNet backbone.**
| Method | Dice(ET) | Dice(WT) | Dice(TC) | Hausdorff(ET)| Hausdorff(WT)| Hausdorff(TC)|
|:------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|
| Scratch | 64.96±0.30 | 83.03±0.17 | 71.34±0.39 | 7.635±0.162 | 10.602±0.504 | 10.942±0.186 | 	 	 
| Full | 68.49±0.20 | 85.56±0.05 | 75.12±0.56 | 6.672±1.492 | 7.878±0.155 | 10.525±1.091 |
| Head | 65.71±0.04 | 84.19±0.08 | 74.77±0.29 | 6.128±0.298 | 7.505±0.036 | 7.864±0.324 |
| VPT-Shallow | 66.02±0.11 | 84.72±0.14 | 75.84±0.28 | 6.114±0.087 | 7.506±0.020 | 8.471±0.014 |
| VPT-Deep | 67.01±0.22 | 85.14±0.04 | 76.80±0.38 | 6.064±0.174 | 7.717±0.134 | 7.648±0.094 |
| Adapter | 68.30±0.12 | 85.37±0.15 | 77.05±0.42 | 5.501±0.031 | 7.636±0.053 | 7.986±0.305 |
| AdaptFormer | 65.88±0.16 | 84.34±0.14 | 74.77±0.62 | 6.652±0.568 | 8.204±0.030 | 8.430±0.232 |
| Pro-tuning | 67.18±0.16 | 85.32±0.04 | 76.51±0.24 | 5.805±0.014 | 7.073±0.266 | 7.564±0.262 |
| ST-Adapter | 69.18±0.19 | 86.27±0.04 | 79.18±0.07 | 6.077±0.043 | 6.939±0.023 | 6.778±0.278 |
| Ours | 70.53±0.12 | 86.58±0.05| 79.35±0.06 | 5.862±0.030 | 6.224±0.051 | 6.947±0.118 |

**Table 9: Performance comparison on MSD with SwinUNETR backbone.**
| Method | Task02 Heart Average Dice | Task06 Lung Average Dice | Task09 Spleen Average Dice
|:------|:-------:|:-------:|:-------:|
| Scratch | 91.95±1.64 | 65.82±4.16 | 95.76±0.52
| Full | 93.73±1.53 | 67.69±4.22 | 96.52±0.33
| Ours | 95.84±1.26 | 78.09±4.04 | 97.06±0.40

> [**Q4**]. The authors have clearly put a lot of effort in preparing the experiments and designing their solution but the numerous language issues throughout the papers make it very difficult to follow. There are many nominal statements without verbs (e.g : Secondly, improve the flexibility and lightness of the framework on p4) Some of the sentences are very convoluted and missing some words References to future results (section 4.2 referred to in section 3.2) does not help with clarity.

[**A4**]. Thanks for your valuable advice, we have further improved certain inappropriate expressions in our manuscript, enhancing the writing quality of our paper. 
For example:
| Before | After | 
|:------|:-------|
|Secondly, improve the flexibility ... | Secondly, sole insertion in encoder part improves the flexibility of the whole framework. Our inserting strategy broadens the adaptability of Med-Tuning on visual foundation models while reducing tuned parameters.|
|... results of Head in Sec. 4.2 show.  | Inadequate feature extraction will hinder performance even with a same robust decoder, as evidenced by the decline in results for Head tuning compared to Full fine-tuning, detailed in Tables 1, 2, and 3.|
We sincerely appreciate your recognition of the novelty and contributions of our paper, as well as the provided positive comments on our extensive experiments and clear identification of the current gaps. We have addressed all of your questions in detail in the following Response section and will incorporate all feedback in the final version. We genuinely hope that our detailed explanations and the additionally supplemented experiments with our best efforts will give us the precious opportunity to raise the evaluation score of our work in your perspective.

**Our Responses to Paper Weaknesses:**

> [**Q1**]. The overall clarity of the paper is quite poor and it becomes difficult to follow the rationale behind the methods.

[**A1**]. Following your valuable suggestions, along with feedback obtained from several additional scholars invited to read this work, we have enhanced the overall clarity of the paper, making it easier to follow and understand. The corresponding revisions will be incorporated into the final version of the paper upon acceptance.




> [**Q2**]. The link between the equations and the text is sometimes unclear. For instance, when reading the description of the Adapter, a natural translation would be $\text{Adapter}(X) = W_{up}(\sigma(W_{down}(X)))$. It is quite unclear where the addition term X+ comes from from the textual description.

[**A2**]. Thanks for your detailed review. According to the definition of the Adapter in [1], the textual description in the paper should be based on formula(1): $\text{Adapter}(X) = X + W_{up}(\sigma(W_{down}(X)))$. A vanilla Adapter contains a skip-connection, allowing the adapter module to approximate an identity function when the parameters of the projection layers are close to zero.

[1] Parameter-efficient transfer learning for NLP. ICML, 2019.
> [**Q2**]. Another point that was a bit surprising to me was the lack of difference in performance shown in the ablation study. If I understood correctly the first line of Table 4 represents no presence of Med-Adapter, and the last line is your proposed method. Although the average values of your performances are superior, I wish the authors presented standard errors to ensure no to little overlap of the confidence intervals to effectively conclude on an improvement. 

[**A2**]. Thanks for this helpful suggestion. Indeed, the first row of Table 4 (in our manuscript) represents the results without the Med-Adapter (i.e., full fine-tuning). Per your advice, we supplement results with standard deviations, which are listed below in Table 5 for your convenience to check. It can be observed that compared to full fine-tuning, Med-Tuning (default) has achieved improvements of **+0.44**, **+1.00**, **+3.18** in **Dice_ET**, **Dice_WT**, **Dice_TC**, and **-1.02**, **-1.18**, **-1.34** in **HF_ET**, **HF_WT**, **HF_TC**, respectively. Meanwhile, it has demonstrated smaller standard deviations in the results, indicating that our proposed Med-Tuning (default) exhibits lower randomness.

**Table 5: Ablation study on the position of inserted Med-Adapter.**
| n=0 | n=1 | n=2 | n=3 | Dice(ET) | Dice(WT) | Dice(TC) | Hausdorff(ET)| Hausdorff(WT)| Hausdorff(TC)| |
|:------|:------|:------|:------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|
-|-|-|-|78.07±0.28 |88.68±0.19 |77.26±0.40 |5.02±0.32 |6.70±0.51 |7.10±0.29
√|√|-|-|74.83±0.30 |87.09±0.27 |72.94±0.44 |7.26±0.37 |13.12±0.60 |10.17±0.41
√|√|√|-|75.60±0.31 |86.79±0.23 |73.41±0.39 |8.44±0.49 |12.32±0.53 |11.24±0.33
√|√|√|√|78.51±0.27 |89.68±0.22 |80.44±0.34 |4.00±0.34 |5.52±0.54 |5.76±0.30 |(default)


> [**Q3**]. One last comment I would have is on the concept itself of PET. I would be curious to see how your method fares compared to training a nnUnet as an even simpler baseline. To my knowledge, nnUnet has always been shown to outperform other methods while being much lighter. Fine-tuning is an interesting process when very little data is present for training and testing but in the datasets you have here, such verification would be possible and empower your claim to use PET methods over retraining a model (albeit much lighter) from the ground up. 

[**A3**]. Many thanks for this kind comment. It's worth noting that, on one hand, some compelling experiments in this paper have already been conducted using the nnU-Net training framework. As mentioned in the appendix, for the experiments on KiTS 2019 dataset, the data augmentations employed are in line with the prior work nnU-Net. In other words, the experiments carried out on the KiTS 2019 within this paper are based on the nnU-Net training framework. The results from experiments on KiTS 2019 clearly show that our method over various baselines (i.e., SwinUnet and ViT) under the nnU-Net framework have achieved significant improvements compared to other tuning methods. This represents a model performance enhancement that is independent of the training framework, sufficiently demonstrating the advantages of our method as well as the superiority of nnU-Net's data preprocessing and training framework. 

On the other hand, following your valuable suggestions, we have also endeavored, during the short time period of the rebuttal, to test the effectiveness of our Med-Tuning using nnU-Net as the baseline model. Based on the results obtained during this short period, the performance after combining nnU-Net with our Med-Tuning framework is very close to the reproduced performance of nnU-Net as the baseline model on KiTS 2019 dataset.

Due to the constraints of time, we were unable to further enhance performance using our Med-Tuning. However, the comparative outcomes have sufficiently demonstrated the universality of our proposed method, without leading to any deterioration in performance. Moreover, there exists potential for achieving higher performance gains. We are optimistic about the future possibilities of our methodology and are committed to exploring these opportunities.
We sincerely appreciate your recognition of our extensive literature review, experiments, and good writing. We have addressed all of your questions in detail in the following Response section and will incorporate all feedback in the revision. We hope that our detailed explanations will give us the precious opportunity to raise the evaluation score of our work in your perspective.

**Our Responses to Paper Weaknesses:**

> [**Q1**]. Although the performances shown are superior to previous methods, I would have liked to see confidence intervals/standard errors of the performances as the results between the methods are very close. If there is an overlap of the results, one could not confidently conclude that such method performs best.

[**A1**]. 
Thanks for this helpful suggestion. Per your advice, we supplement some representative raw results accompanied by their standard deviations, which are listed below in Table 1-4 for your convenience to check. The results indicate that, despite **minor overlaps in certain cases**, our method is **generally comparable to or surpasses** the results of full fine-tuning and most of the previous PET methods.

**Table 1: Performance comparison on KiTS 2019 with SwinUnet backbone.**
| Method | Kidney Dice | Tumor Dice | 
|:------|:-------:|:-------:|
| Scratch | 94.33±0.06 | 61.10±0.40 |	 
| Full | 94.68±0.07 | 62.13±0.97 |
| Head | 91.95±0.03 | 53.93±0.35 |
| VPT-Shallow | 91.72±0.09 | 54.86±0.71 |
| VPT-Deep | 91.53±0.12 | 53.41±0.76 |
| Adapter | 93.02±0.04 | 57.15±0.62 |
| AdaptFormer | 93.74±0.03 | 59.79±0.98 |
| Pro-tuning | 90.34±0.08 | 51.19±0.46 |
| ST-Adapter | 92.97±0.07 | 57.33±0.53 |
| Ours | 95.69±0.05 | 70.14±0.54 |

**Table 2: Performance comparison on BraTS2019 with SwinUnet backbone.**
| Method | Dice(ET) | Dice(WT) | Dice(TC) | Hausdorff(ET)| Hausdorff(WT)| Hausdorff(TC)|
|:------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|
Scratch|78.38±0.29|88.59±0.12|76.46±0.45|6.055±0.202|10.651±0.531|9.176±0.301|
Full|78.26±0.25|89.56±0.07|79.16±0.63|4.327±0.598|6.149±0.179|6.704±0.120|
Head|78.07±0.07|88.68±0.09|77.26±0.33|5.021±0.374|6.697±0.082|7.091±0.392|
VPT-Shallow|77.16±0.18|88.30±0.15|76.77±0.27|5.421±0.108|6.154±0.044|7.345±0.053|
VPT-Deep|77.02±0.22|88.65±0.07|76.91±0.32|5.297±0.220|7.088±0.157|7.940±0.132|
Adapter|77.98±0.25|89.22±0.19|78.02±0.46|5.298±0.061|6.622±0.094|8.490±0.359|
AdaptFormer|77.69±0.13|88.61±0.14|76.83±0.68|4.909±0.802|6.290±0.058|7.885±0.287|
Pro-tuning|78.58±0.14|89.33±0.04|78.79±0.36|5.273±0.040|6.410±0.323|8.236±0.376|
ST-Adapter|78.40±0.19|89.54±0.02|77.44±0.11|4.751±0.068|6.013±0.037|7.405±0.317|
Ours|78.51±0.16|89.68±0.03|80.44±0.12|4.003±0.062|5.517±0.043|5.756±0.304|

**Table 3: Performance comparison on BraTS2019 with ViT+UPerNet backbone.**
| Method | Dice(ET) | Dice(WT) | Dice(TC) | Hausdorff(ET)| Hausdorff(WT)| Hausdorff(TC)|
|:------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|
| Scratch | 64.96±0.30 | 83.03±0.17 | 71.34±0.39 | 7.635±0.162 | 10.602±0.504 | 10.942±0.186 | 	 	 
| Full | 68.49±0.20 | 85.56±0.05 | 75.12±0.56 | 6.672±1.492 | 7.878±0.155 | 10.525±1.091 |
| Head | 65.71±0.04 | 84.19±0.08 | 74.77±0.29 | 6.128±0.298 | 7.505±0.036 | 7.864±0.324 |
| VPT-Shallow | 66.02±0.11 | 84.72±0.14 | 75.84±0.28 | 6.114±0.087 | 7.506±0.020 | 8.471±0.014 |
| VPT-Deep | 67.01±0.22 | 85.14±0.04 | 76.80±0.38 | 6.064±0.174 | 7.717±0.134 | 7.648±0.094 |
| Adapter | 68.30±0.12 | 85.37±0.15 | 77.05±0.42 | 5.501±0.031 | 7.636±0.053 | 7.986±0.305 |
| AdaptFormer | 65.88±0.16 | 84.34±0.14 | 74.77±0.62 | 6.652±0.568 | 8.204±0.030 | 8.430±0.232 |
| Pro-tuning | 67.18±0.16 | 85.32±0.04 | 76.51±0.24 | 5.805±0.014 | 7.073±0.266 | 7.564±0.262 |
| ST-Adapter | 69.18±0.19 | 86.27±0.04 | 79.18±0.07 | 6.077±0.043 | 6.939±0.023 | 6.778±0.278 |
| Ours | 70.53±0.12 | 86.58±0.05| 79.35±0.06 | 5.862±0.030 | 6.224±0.051 | 6.947±0.118 |

**Table 4: Performance comparison on MSD with SwinUNETR backbone.**
| Method | Task02 Heart Average Dice | Task06 Lung Average Dice | Task09 Spleen Average Dice
|:------|:-------:|:-------:|:-------:|
| Scratch | 91.95±1.64 | 65.82±4.16 | 95.76±0.52
| Full | 93.73±1.53 | 67.69±4.22 | 96.52±0.33
| Ours | 95.84±1.26 | 78.09±4.04 | 97.06±0.40