One-to-Multiple: A Progressive Style Transfer Unsupervised Domain-Adaptive Framework for Kidney Tumor Segmentation

Authors: kai  hu
jinhao  li
yuan  zhang
xiongjun  ye
xieping  gao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:03:58

Abstract: In multi-sequence Magnetic Resonance Imaging (MRI), the accurate segmentation of the kidney and tumor based on traditional supervised methods typically necessitates detailed annotation for each sequence, which is both time-consuming and labor-intensive. Unsupervised Domain Adaptation (UDA) methods can effectively mitigate inter-domain differences by aligning cross-modal features, thereby reducing the annotation burden. However, most existing UDA methods are limited to one-to-one domain adaptation, which tends to be inefficient and resource-intensive when faced with multi-target domain transfer tasks. To address this challenge, we propose a novel and efficient One-to-Multiple Progressive Style Transfer Unsupervised Domain-Adaptive (PSTUDA) framework for kidney and tumor segmentation in multi-sequence MRI. Specifically, we develop a multi-level style dictionary to explicitly store the style information of each target domain at various stages, which alleviates the burden of a single generator in a multi-target transfer task and enables effective decoupling of content and style. Concurrently, we employ multiple cascading style fusion modules that utilize point-wise instance normalization to progressively recombine content and style features, which enhances cross-modal alignment and structural consistency. Experiments conducted on the private MSKT and public KiTS19 datasets demonstrate the superiority of the proposed PSTUDA over comparative methods in multi-sequence kidney and tumor segmentation. The average Dice Similarity Coefficients are increased by at least 1.8% and 3.9%, respectively. Impressively, our PSTUDA not only significantly reduces the floating-point computation by approximately 72% but also reduces the number of model parameters by about 50%, bringing higher efficiency and feasibility to practical clinical applications.

Main paper: 
 introduction: 
 Kidney tumor segmentation is a crucial task in medical image analysis, playing an essential role in the diagnosis, staging, and treatment of kidney cancer [1, 2, 3] . Previous studies have primarily focused on computed tomography (CT) images [4, 5, 6, 7, 8, 9] , but in recent years, magnetic resonance imaging (MRI) has emerged as a safer alternative due to its non-radiative nature [10, 11, 12] . Compared to CT, MRI offers superior contrast for characterizing kidney tissue and tumors, enabling a clearer distinction between normal and pathological tissues. Furthermore, the multi-parameter imaging capability of MRI generates diverse sequences [13, 14, 15] that enhances the comprehensive description of pathological features. These advantages have established MRI as the preferred modality for clinicians in diagnosing kidney disease, particularly kidney tumors, and have facilitated the widespread use of multi-sequence MRI in clinical practice for kidney and tumor segmentation.In existing studies, traditional supervised segmentation methods typically rely on a large amount of annotated data, and performing fine-grained annotations for a single sequence is already a timeconsuming and labor-intensive task, let alone for every sequence. Unsupervised domain adaptation (UDA) methods [16, 17, 18, 19, 20] have been proposed as a promising solution to address these challenges. Current UDA methods mainly focus on one-to-one domain adaptation, involving a single source domain and a target domain. Although these methods perform well in certain scenarios, they lack scalability, meaning that for multi-target domain tasks, training paired generators for each sourcetarget domain pair results in significant computational costs and resource consumption. Furthermore, one-to-one domain adaptation methods can only learn fixed mappings between two domains, thereby failing to capture the potential commonalities and connections among multiple sequences.As an initial attempt to apply multi-domain adaptation technology to medical imaging, Xu et al. [21] proposed OMUDA for abdominal organ segmentation using multi-sequence MRI. However, the architecture of OMUDA is derived from StarGAN v2 [22] , which was originally designed for style transfer in natural images and has not been specifically tailored and optimized for the needs of domain adaptation in medical imaging. Although it shows improvement in resolving domain confusion across multiple domains, this limitation leads to suboptimal generated images with regards to organ structural consistency and detail preservation. Consequently, the task of one-to-multiple domain adaptation in medical imaging remains a significant challenge, necessitating more refined and specialized approaches for further advancement.In this paper, we propose a One-to-Multiple Progressive Style Transfer Unsupervised Domain-Adaptive (PSTUDA) framework, which explicitly stores multi-level style features from different domains in designated multi-level style dictionaries, thus alleviating the burden on the generator and achieving the decoupling of content features from style features. Our PSTUDA employs multiple cascaded style fusion modules to recombine multi-level style features from different domains with content features layer by layer using Point-wise Instance Normalization (PIN), thereby ensuring that the generated images across multiple target domains have high-quality style and structure.The main contributions of our work are summarised as follows:• We explore a novel and efficient One-to-Multiple Progressive Style Transfer Unsupervised Domain-Adaptive (PSTUDA) framework, which is capable of simultaneously transferring a single annotated source domain to multiple unannotated target domains, significantly reducing the need for tedious domain adaptation work for each target domain. • We introduce a multi-level style dictionary that stores style information for each domain at different stages of style transfer, alleviating the burden on the generator to perform multiple tasks and effectively decoupling content features from style features. • We propose a progressive style transfer paradigm and a Point-wise Instance Normalization (PIN) method. The former comprises multiple cascading style fusion modules, each recombining content features with corresponding style features through PIN, thereby achieving better cross-modal alignment and structural consistency. • We construct a multi-sequence kidney tumor MRI dataset called MSKT to facilitate research on kidney tumor analysis. Quantitative and qualitative results on the MSKT and the public dataset KiTS19 show that our PSTUDA framework outperforms the state-of-the-art methods and significantly improves segmentation performance and training efficiency.2 Related Work 
kidney tumor segmentation: 
 Computer-aided diagnostic methods for kidney tumor segmentation play a crucial role in clinical practice [23, 24, 25, 26, 27] . The significant variability in the size, shape, and location of kidneys and tumors presents a considerable challenge to accurate segmentation. Yu et al. [28] developed Crossbar-Net to capture global and local features of kidney tumors through crossbar patches and focused on difficult-to-segment regions through a cascade training strategy. Myronenko et al. [29] designed a dedicated boundary branch supervised by an edge-aware loss term to enhance the consideration of organ and tumor edge information. These approaches are centered on improving the ability of the model to recognize the complex morphology of kidney tumors. To further advance this effort, Pandey et al. [30] integrated active contouring with 3D-U-Net to achieve precise delineation of kidney tumor shapes, showcasing the potential of merging deep learning with traditional image processing techniques. However, these approaches typically rely on fully supervised learning with extensive pixel-level annotations. To address this limitation, researchers are exploring alternative solutions in the fields of semi-supervised, self-supervised, and unsupervised learning. Ma et al. [31] introduced an Affinity Network that learns from limited data using k-nearest neighbors attention pooling layers. Similarly, Ciga et al. [32] and Faust et al. [33] developed methods that enhance feature learning and guide tumor analysis through self-supervised and unsupervised techniques, respectively. By reducing the reliance on annotated data, these methods not only lower costs but also improve the model's generalization capabilities.UDA is one of the important methods for addressing domain difference. It aims to transfer a model from an annotated source domain to an unannotated target domain. Existing works have mostly focused on one-to-one domain adaptation [34, 35, 36, 37, 38, 39] , yielding impressive outcomes. For example, CycleGAN [34] used cycle consistency constraints to transform unpaired images from one domain to another. CyCADA [40] enforced cycle consistency by combining methods of image space alignment and latent representation space alignment. MUNIT [41] decomposed image representations into content and style codes, enabling multimodal image translation. For medical image domain adaptation, SIFA [35] achieved domain alignment from both image and feature perspectives, enabling the segmentation network to adapt to the unannotated target domain. Thereafter, DEPL [39] further improved the segmentation accuracy by employing multi-source domain extension and selective pseudo-labelling strategies.However, these one-to-one domain adaptation methods lack scalability when handling multiple domain transfer tasks, as they can only learn the relationships between two different domains at a time. StarGAN [42] performed image-to-image translation across multiple domains using a single model, and then the improved version, StarGAN v2 [22] further enhanced diversity in generated images by introducing style codes specific to each domain. Additionally, Sharma and Hamarneh [43] proposed a multi-modal generative adversarial network that leveraged redundant information from available sequences to synthesize missing MRI pulse sequences in patient scans. Gholami et al. [44] developed an information-theoretic approach that aimed to find shared latent spaces for domain adaptation across multiple target domains. It should be noted that these methods predominantly focus on the diversity of the generated images, while paying less attention to the structural consistency before and after image translation.In UDA tasks, image normalization is a key step that can help models to learn and transfer features effectively [45, 46, 47, 48, 49, 50] . Instance Normalization (IN) [51] normalizes the features of each sample to improve the quality and realism of generated images. Adaptive Instance Normalization (AdaIN) [52] dynamically adjusts the relationship between style and content features of input images to enable a rapid transformation across arbitrary styles. Batch-Instance Normalization (BIN) [53] explicitly normalizes unnecessary style variations in images while preserving useful styles. AdaAttN [54] introduces adaptive attention normalization to optimize the effects of arbitrary neural style transfer. GramLIN [55] continuously measures the proximity of the current stylized output to the target style to achieve progressive stain transfer. In medical image domain adaptation tasks, it is crucial to maintain structural consistency during image translation, in addition to changing image styles. Therefore, we propose a novel normalization method called PIN, which progressively fuses content and style features at each pixel by considering image details and local style differences. The goal is to ensure well-structured images after transfer. Formally, for a one-to-multiple domain adaptation task, we have one source domain S and multiple target domains T i . The source domain S is annotated, denoted as S = {(x j , y j )} Ns j=1 , while the target domains are unannotated, denoted asT i = {x Ti j } N T ij=1 . Each domain is assigned a domain label, for example, L = [0, 1, ..., N ] representing the source domain 0, target domain 1, etc., up to target domain N. Figure 1 (a) illustrates our PSTUDA framework, which mainly consists of a shared generator (including an encoder, a decoder, and multiple style fusion modules) and a multiscale discriminator. X src , X trgs , and X recon represent the input image from the source domain, the generated pseudo-target domain images corresponding to the input image, and the image reconstructed back to the source domain from the pseudo-target domain images, respectively. Our task involves two stages. The goal of the first stage is to train a single generator G such that given a source domain image X src ∈ S and any target domain label l i , it can generate a pseudo-target domain image X trg_i corresponding to the image X src , i.e., G(X src , l i ) ⇒ X trg_i . The objective of the second stage is to utilize the generated pseudo-target domain images X trgs and source domain annotation y to train a segmentation network to achieve accurate segmentation of the kidney and tumor. 
multi-level style dictionary: 
 Our design is inspired by the StarGAN v2 multi-domain image generation framework, which employs a Mapping Network and a Style Encoder to obtain style encodings. The Mapping Network transforms random Gaussian noise into diverse style encodings, while the Style Encoder extracts style encodings from reference images. These techniques are essential for transforming natural image styles and enhancing artistic variety. However, in medical image domain adaptation, they encounter challenges in extracting comprehensive style features. Style encodings from a single image may not capture the full range of styles within a domain, and deriving styles from Gaussian noise, while introducing diversity, can result in instability. In medical image analysis, such instability could potentially compromise feature recognition and segmentation accuracy.To address these challenges, we propose to define a set of learnable multi-level style dictionaries for each domain, where the style information at each level closely corresponds to the respective phase of style fusion. This multi-level structure allows for starting with basic style features and exploring more complex layers of style, thereby providing a progressively refined path for the style fusion process. During the generative adversarial process, the continuous updating and learning of the style dictionary can adapt to the ever-changing style demands. The early-stage stylized features will provide feedback for subsequent style feature learning, thus guiding them to update in a dynamic and targeted manner. By learning style information from the whole domain, the style dictionary becomes more representative, which effectively overcomes the limitation of extracting style encoding from a single image. Moreover, the iterative updates of style dictionaries enhance stability and reduce uncertainty in extracting style encodings from random noise. Storing style encoding information in a multi-level dictionary helps alleviate burden on the generator and achieves effective decoupling of content features and style features. This allows it to focus on capturing domain-invariant features such as structure and shape. In this way, the multi-levels style dictionary refines complex styles within target domains at each level, ensuring that each can timely reflect latest progress in style transfer process more precise and coordinated.In PSTUDA, multiple cascaded style fusion modules and decoders constitute the core components of progressive style transfer. Figure 1 (b) illustrates the main processes of the first and last (decoding phase) style transfer stages, with similar style transfer processes in the intermediate stages. In the first style transfer stage, the style fusion module requires two inputs: the content feature X 1 ∈ R H×W ×C obtained by downsampling from the encoder, and the first-level style encoding V 1 ∈ R S×S×1 from the target domain, matched with the current style fusion module. We utilize the domain label L multiplied by the first style dictionary to select the target style encoding V 1 . First, the content feature X 1 passes through a convolutional layer to obtain the content feature X1 , and the target style encoding V 1 undergoes two consecutive 1 × 1 convolutions for channel transformation to obtain the style feature V1 ∈ R H×W ×C .After obtaining the content feature X1 and style feature V1 , we propose a novel style fusion normalization method, Point-wise Instance Normalization (PIN), to effectively combine the two for more subtle and accurate pixel-level style transfer. The PIN can be defined as:EQUATIONEQUATIONEQUATIONwhere N orm(•) denotes the Instance Normalization of the content feature Xl ∈ R H×W ×C at layer l. The parameters γ chw and β chw are scaling and shifting parameters specific to the target domain. These parameters are derived by applying ConvBlock and Chunk operations to the style feature Vl ∈ R H×W ×C and are adjusted to match the spatial dimensions of the content feature Xl , thus enabling independent style transfer at each pixel. By rescaling the feature map using these parameters, style information specific to the target domain is integrated into the feature map for style transfer. PIN provides unique style statistics for each spatial point of the content feature, allowing for customized style fusion based on different regions and features of the image content. This facilitates finer local style variations and richer style details, making it particularly suitable for applications requiring precise style adjustments, such as medical image segmentation.In the final stage of style transfer, the decoder takes the stylized content feature X n ∈ R H ′ ×W ′ ×C ′ from the previous layer and the last-level style encoding V n ∈ R S×S×1 as input. To spatially match the content feature X n , the style encoding V n first undergoes upsampling through a deconvolutional layer for scale transformation, followed by two consecutive 1 × 1 convolutions for channel transformation, resulting in the style feature Vn ∈ R H ′ ×W ′ ×C ′ . The other style fusion operations are similar to those in the first stage. There are two considerations for performing style transfer during the decoding phase. Firstly, the decoding phase is the process of image reconstruction, where integrating style features can effectively incorporate high-level style information from the target domain into the image. This helps to better express the target style while preserving content information, especially in terms of details and textures. Secondly, as the decoder is responsible for upsampling, it has the opportunity to refine and restore image details while enlarging the feature maps. Introducing style transfer at this stage ensures that these details not only conform to the content structure but also match the characteristics of the target style, thus achieving effective fusion of content and style at different scales.Generator As previously mentioned, the generated pseudo-target domain images will be used in conjunction with the corresponding segmentation annotations from the source domain to train the subsequent segmentation networks. Therefore, it is crucial to maintain structural consistency during the image translation process for the success of downstream applications. The generator in StarGAN v2, due to its multiple downsampling steps, unfortunately leads to a loss of spatial information, which poses challenges in maintaining structural consistency during image translation. In light of CycleGAN's outstanding performance in one-to-one medical image translation, we adopt the approach of OMUDA by integrating the generator architecture of CycleGAN into our encoder and decoder. To facilitate a single generator to handle one-to-multiple transfer tasks, we replace the IN layers in the decoder with PIN layers. The complete generator comprises an encoder, a decoder, and a series of cascaded style fusion modules. Among them, the encoder, equipped with two downsampling layers and multiple IN layers, is tasked with extracting domain-invariant features from the input image of the source domain. The cascaded style fusion modules are responsible for integrating the style features of the target domain with the domain-invariant features of the source domain. The decoder is composed of two upsampling layers embedded with PIN and is responsible for style transfer and image reconstruction during the decoding phase.Discriminator Inspired by Wang et al.'s work [56] , we enhance the original discriminator architecture of StarGAN v2 with a multi-scale mechanism. As depicted in Fig. 2 , the discriminator is composed of multiple independent discrimination branches, each containing four residual blocks [57] , with each residual block having N output branches serving N specific target domains. The input to the discriminator includes not only the original image but also images processed through different scales of downsampling. The multi-branch output of the discriminator first multiplies with the target domain label L to select the discrimination output for the corresponding domain before proceeding to authenticity determination. By evaluating the generated images at various scales, the multi-scale discriminator can more comprehensively judge the realism of the images, thereby encouraging the generator to produce images with higher quality. More detailed information on the generator and discriminator architectures can be found in Appendix B.1.For convenience, we give the following symbol definition. x s ∈ X denotes a source domain image, with its corresponding source domain label l s ∈ L. l t ∈ L represents any target domain label. G and D stand for the generator and discriminator, respectively. The loss functions in the generator include adversarial loss L adv , cycle consistency loss L cyc , and identity loss L idt [58] . The loss in the discriminator includes adversarial loss L adv . Among them, the adversarial loss is defined asEQUATIONwhich encourages the generator to generate images that are indistinguishable from the target domain images by the discriminator. The cycle consistency loss is computed asEQUATIONwhich ensures that an image translated from the source domain to the target domain can be translated back to the original image, thus preserving the structural consistency of the image throughout the translation process. The identity loss can be defined asEQUATIONwhich is used to preserve the content of the source domain images when the generator is applied to them. The complete training loss can be summarized as follows:EQUATIONwhere λ cyc and λ idt denote the weights for the corresponding loss terms and they are set to 10 and 1 in our experiments, respectively.4 Experiments 
dataset: 
 In this study, we utilize two datasets for the performance evaluation of the method. The first one is a private dataset named MSKT, which comprises 104 cases, each including four sequences: T1c, FS T2W, T2W, and DWI. The second is the publicly available KiTS19 [59] dataset. We conduct our first set of comparative experiments using the MSKT dataset. Specifically, the annotated T1c data is served as the source domain, while the unannotated FS T2W, T2W, and DWI data are considered as the target domain. For our second set of comparative experiments, we combine KiTS19 with the MSKT dataset. In this case, all training data from KiTS19 are used as the source domain, and all four sequences of the MSKT are considered as the target domain. We ensure that there is no overlap between the source domain, target domain, and test set to prevent any potential information leakage. More details about the dataset and evaluation metrics are provided in Appendix A.To comprehensively evaluate the performance of the proposed method, we compare PSTUDA with five other state-of-the-art UDA methods on the MSKT and KiTS19 datasets: CycleGAN [34] , MUNIT [41] , SIFA [35] , DEPL [39] , and StarGAN v2 [22] . Except for SIFA and DEPL, the others are limited to cross-domain image transfer. Therefore, we train a dedicated U-Net [60] from scratch based on the pseudo-target domain images generated by these methods for multiple sequences. For a fair comparison, all hyperparameters in the U-Net remain consistent. Due to the one-to-one image transfer characteristics of SIFA, DEPL, CycleGAN, and MUNIT, we train a separate model for each sequence using these methods to handle multi-sequence MR image transformations.Due to the unpaired nature of multi-sequence MR images, it is challenging to quantitatively assess the style effects and structural consistency of the generated images. Experience suggests that if anatomical structures are distorted during the image transfer process, the pseudo-target domain images generated will not correspond with the annotations of the source domain, thereby impairing segmentation performance. Moreover, if there is a significant difference between the style of the generated images and the real target domain images, this discrepancy in data distribution may also detrimentally affect the segmentation results. Hence, the quality of the generated images is indirectly indicated by the segmentation results. We also observe that PSTUDA significantly surpasses MUNIT, SIFA, DEPL, and StarGAN v2 from both DSC and HD 95 . Although SIFA and DEPL are more efficient end-to-end domain adaptation segmentation methods, they do not perform well in our task. The images generated by these methods exhibit structural distortions and lack naturalness (4 th and 5 th columns in Fig. 3 ). Particularly, DEPL shows severe structural distortion of kidneys and tumors in the T2W sequence, which has greater inter-domain differences. MUNIT and StarGAN v2, while excelling in the diversity of generated images, fall short in maintaining cross-domain structural consistency, leading to distorted structures in the generated images (3 rd and 6 th columns in Fig. 3 ). The performance of CycleGAN is close to that of PSTUDA on FS T2W and DWI sequences, but it underperforms on the T2W sequence, as the translated images lose many important structural details, resulting in poor image alignment.In contrast, our PSTUDA prioritizes structural consistency in the cross-domain image translation process by separating content from style and progressively recombining content and style features in a point-wise manner. This approach produces images with higher fidelity, as demonstrated in the last column of Fig. 3 , thereby significantly enhancing segmentation performance. Results on KiTS19 and MSKT This experiment utilizes CT images from KiTS19 as the source domain and four MRI sequences from MSKT as the target domain. Given the different imaging technologies employed by CT and MRI, the domain difference between them is significantly greater than that between the various MRI sequences. As shown in Table 2 , most comparative methods show a reduction in kidney and tumor segmentation performance on FS T2W, T2W, and DWI when compared to the first set of experiments. Although the average DSC of CycleGAN on the DWI is higher than that of PSTUDA (by about 1.9%), its performance is lower than that of PSTUDA by approximately 7.5% and 9.1% on FS T2W and T2W, respectively. Our PSTUDA almost achieves the best average DSC results on all sequences. Compared to the first set of experiments, the performance of CycleGAN has decreased by about 6.8% and 4.4% on FS T2W and T2W, respectively, while the performance of the proposed PSTUDA has decreased by 0.5% and increased by 1.2% on these two sequences. These results underscore PSTUDA's exceptional robustness in tasks with larger domain differences. As shown in Fig. 4 in Appendix C, our PSTUDA outperforms other comparative methods in maintaining structural consistency in translated images. Moreover, thanks to the one-to-multiple features, our method also shows a significant advantage in training efficiency, which will be detailed in the subsequent section. In summary, the highlight of PSTUDA lies in its dual improvement of segmentation performance and training efficiency. Training efficiency In this section, we evaluate the performance of PSTUDA in comparison with CycleGAN, MUNIT, and StarGAN v2 in terms of model parameters and FLOPs. As shown in Table 9 in Appendix D, the FLOPs for CycleGAN and MUNIT represent the cumulative results of the image translation tasks for the three domains, i.e., T1c→FS T2W, T1c→T2W, and T1c→DWI.For the transfer of T1c to the other three MR sequences using CycleGAN, MUNIT, and StarGAN v2, the parameters that need to be optimized are 2, 3.3, and 1.9 times that of PSTUDA, respectively. Regarding FLOPs, PSTUDA demonstrates a reduction of 72% and 79% compared to CycleGAN and MUNIT, respectively, but shows a 17% increase compared to StarGAN v2. The increased computational demand is mainly due to the PIN module in PSTUDA, which requires per-pixel fusion of style and content features. It is worth noting that the training efficiency of PSTUDA becomes more advantageous when more target domains are included in a one-to-multiple domain adaptation task. Overall, PSTUDA substantially decreases both model parameters and FLOPs while maintaining high segmentation accuracy, implying lower memory requirements and faster inference speed. In summary, these results highlight the significant advantages of PSTUDA in enhancing the efficiency and feasibility of clinical applications.Effectiveness of PST and MS_D To investigate the effectiveness of the proposed Progressive Style Transfer (PST) paradigm, we conduct ablation studies on it with the Multi-Scale Discriminator (MS_D) on the MSKT dataset. As shown in Table 3 , the introduction of PST significantly improves the DSC and HD 95 metrics compared to the Baseline. This reflects the ability of the PST to enhance the realism of translated images while maintaining structural consistency. Furthermore, the multiscale discriminator assesses the pseudo-target domain images at different scales, thereby further enhancing the capabilities of the generator.We also conduct ablation studies on the internal sub-modules within PST, namely the Multi-level Style Dictionary (M_SD) and the Point-wise Instance Normalization (PIN). As illustrated in Table 4 , replacing the Mapping Network and Style Encoder in the Baseline with the M_SD results in a significant performance improvement. This can be attributed to the stability and representativeness of the learned style encodings. Moreover, the integration of the M_SD with PIN fully considers the details and local style variations of images, thereby yielding exceptional results. Ablation of PIN and other normalization methods We compare PIN with AdaIN [52] and BIN [53] , and as shown in Table 5 , PIN outperforms the others. We attribute this superior performance to PIN's ability to account for local style differences, which is particularly advantageous for finegrained segmentation tasks (e.g., kidney tumors, as abnormal pathological tissues, exhibit significant style differences). This capability enables PSTUDA to generate synthetic images that align more closely with the data distribution of the target domain.  
conclusion: 
 In this work, we propose a novel and efficient One-to-Multiple PSTUDA framework that utilizes a multi-level style dictionary to decouple and store style information. By employing multiple cascaded style fusion modules, our framework progressively recombines content and style features, thereby achieving superior cross-modal alignment and consistency of medical tissue structures. Experimental validation on both a private dataset and a public dataset demonstrates the significant advantages of our method in improving training efficiency for one-to-multiple domain adaptation tasks and enhancing the accuracy of multi-sequence kidney tumor segmentation.Table 6 : Data partitioning for each sequence in the MSKT dataset. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work does not pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all the relevant assets used in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. Justification: We provide an explanation about this issue in Appendix A. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [Yes] Justification: Our private dataset has been reviewed and approved by the IRB, and we provide an explanation about this issue in Appendix A. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: CycleGAN This method is derived from the GitHub repository: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix. In our experiments, ResNet_9block and PatchGAN with 3 convolutional layers are selected as the generator and discriminator for CycleGAN, respectively. The weights for adversarial loss (MSE), cycle consistency loss (L1), and domain-invariant perceptual loss (L1) are set to 1, 10, and 5, respectively. Other hyperparameters are consistent with the open-source code. During the segmentation phase, all hyperparameters for U-Net are the same as those in our PSTUDA.MUNIT This method is from the GitHub repository: https://github.com/NVlabs/MUNIT. The architecture of the MUNIT network remains the same as the open-source code. The weights for adversarial loss (MSE), image reconstruction loss (L1), style reconstruction loss (L1), and content reconstruction loss (L1) are set to 1, 10, 1, and 1, respectively. In the segmentation stage, all U-Net hyperparameters are identical to those in our PSTUDA.SIFA This method is available from the GitHub repository: https://github.com/JianghaoWu/SIFApytorch. The official version of this method is implemented in TensorFlow, available at https://github.com/cchen-cc/SIFA. In OMUDA, they have re-implemented SIFA in PyTorch, and we have used the PyTorch version of SIFA to ensure all methods use the same framework for a fair comparison. All settings, hyperparameters, and losses used in our experiments are the same as those in the open-source code.DEPL This method is based on our own implementation.This method is from the GitHub repository: https://github.com/clovaai/stargan-v2.In our experiments, the network architecture is consistent with the official code. The weights for R1 regression loss, adversarial loss (CE), cycle consistency loss (L1), style reconstruction loss (L1), and diversity-sensitive loss (L1) are all set to 1. High-pass filtering is not used in this comparative experiment. During the segmentation phase, all U-Net hyperparameters are the same as those in PSTUDA.In this section, we present the visualized results of the second set of comparative experiments conducted on the KiT19 and MSKT datasets, and these visualizations are discussed in Section 4.2. This section displays the comparative outcomes in terms of model parameters and FLOPs among different cross-domain image translation methods. As illustrated in Table 9 , we compare PSTUDA with CycleGAN, MUNIT, and StarGAN v2, because these methods do not involve a segmentation stage. It is worth note that the style dictionary itself does not contribute to the computation of FLOPs. All models are trained using FP32 precision. The detailed analysis of these comparative results is provided in Section 4.2. To further validate the generalization capability of PSTUDA, we conduct bidirectional cross-modal validation experiments on a publicly available abdominal multi-organ dataset [62] and perform reverse validation experiments from MR to CT on the MSKT and KiTS19 datasets. The results, as shown in Tables 10 and 11 , indicate that our method significantly outperforms StarGAN v2 (baseline) in both experimental groups.The multi-level style dictionary and style fusion module are the core components of PSTUDA, and these two modules include the following hyperparameters: the depth of the style dictionary, the number of levels in the style dictionary, and the number of style fusion modules. It is important to note that the number of levels in the style dictionary is equal to the number of style fusion modules.To investigate the sensitivity of model performance to various hyperparameters, we conduct extensive ablation studies on the depth of the multi-level style dictionary (M_SD) and the number of style fusion modules (SFM). As presented in Tables 12 and 13 , under the original settings (dictionary Limitations Our work presents a domain adaptation method for multi-target domain image translation. However, the field of one-to-multiple domain adaptation is not extensively explored, and our study may have some limitations that should be acknowledged. Firstly, the tasks we focus on are confined to domain adaptation of multi-sequence MR images and the segmentation of kidneys and tumors as presented in this work. In future research, we aim to broaden our scope to include domain adaptation for other medical and natural images, as well as other downstream tasks such as object detection. Secondly, compared to the ideal one-to-multiple model, our current model inherently operates within a broader multi-domain to multi-domain adaptation framework. Therefore, in our future studies, we aspire to design a more specialized one-to-multiple model, which is expected to further improve model performance and reduce model complexity.Broader impacts In fact, the impact of one-to-multiple domain adaptation technology is profound, especially when facing numerous target domains and limited annotated data. Our PSTUDA framework can efficiently extend an annotated source domain to adapt to multiple unannotated new domains, significantly reducing the time and resource investment required for multi-domain transfer tasks. This capability is crucial in complex medical data environments, as it not only greatly alleviates the workload of medical professionals but also holds promise for improving the diagnosis and treatment processes for patients. Overall, our research enhances the flexibility and efficiency of domain adaptation technology, which advances the field of machine learning and brings innovative solutions to critical industries such as healthcare.• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discuss the potential broader impacts of our work in Appendix G. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper receives positive reviewing comments from three reviewers. Two reviewers rate it as 'borderline accept' and a third reviewer rates it as 'strong accept' after rebuttal and discussions. 

The rating by Reviewer tgyp is shown as 'Borderline reject', but in the discussion box, s/he mentioned explicitly that "I suggest changing the score to 5: Borderline Accept." 

Therefore, I recommend 'Accept'.

Reviews: review: paper_summary: paper_summary: The paper presents a one-to-multiple progressive style transfer unsupervised domain adaptation framework designed for kidney and tumor segmentation. 

It aims to mitigate the challenges of annotation burden and domain differences by employing a multi-level style dictionary and cascading style fusion modules. 

It demonstrates significant improvements in segmentation performance and efficiency on the MSKT and KiTS19 datasets.
strength_weakness: strength_weakness: x. **Concern about novelty.**

First, the authors extended the UDA setting to 'one-to-multiple'. I am not sure about its practical or clinical relevance here compared to continued UDA or UDA on evolving domains. 
Second, there seems to be limited novelty compared to the architecture of OMUDA derived from StarGAN v2. 

x. **Concern about the effectiveness of PIN**: 
Point-wise Instance Normalization is a central component of the framework, but its effectiveness compared to other normalization techniques is not thoroughly evaluated. There might be scenarios where the PIN is less effective, and alternative normalization methods like Adaptive Instance Normalization or Batch-Instance Normalization might perform better. 

x. **Concern on analysis.**
The multi-level style dictionary and the cascading style fusion modules. Understanding the contribution of each part would help in refining the model. The sensitivity of the model’s performance to various hyperparameters, such as the temperature parameter (\tao), the number of style fusion modules, and the depth of the style dictionary, is not extensively studied.
questions: questions: Please see my comments above.
limitations: limitations: yes
paper_summary: paper_summary: The authors propose a novel and efficient One-to-Multiple Progressive Style Transfer Unsupervised Domain-Adaptive (PSTUDA) framework to address the UDA task for MRI sequences. Specifically, they developed a multi-level style dictionary that explicitly stores style information for each target domain at different stages, reducing the burden on a single generator in multi-target transfer tasks and effectively decoupling content and style. Additionally, multiple cascaded style fusion modules are employed, utilizing point-wise instance normalization to progressively recombine content and style features, thereby enhancing cross-modal alignment and structural consistency. Experiments conducted on both the private MSKT and public KiTS19 datasets demonstrate the effectiveness of this approach.
strength_weakness: strength_weakness: 1.	The open-source code is not explicitly provided.
2.	The explanation of the One-to-Multiple Framework is somewhat rough and needs more details.
3.	PSTUDA needs to be validated bidirectionally on cross-modal data, such as MRI and CT.
questions: questions: 1.	Reading content from the multi-level style dictionary, does it incur additional overhead? If a new unknown domain is encountered, the learning of the multi-level style dictionary requires a warm-up (there is additional overhead the first time a new domain is encountered).
2.	What is puzzling is that Table 6 shows the multi-level style dictionary only takes up 0.1M; does this include tensors?
3.	Is there any difference between the Generator and Discriminator Architecture in Section 3.4 and previous work?
4.	The theory of achieving independent style transfer for each pixel using Point-wise Instance Normalization (PIN) requires further validation. While it may be effective, the explanation of its application in style transfer is not sufficiently thorough. Additionally, although the targets in medical image segmentation are more precise, this does not necessarily imply that style adjustments need to be more precise. Further explanation is needed.
limitations: limitations: The authors have addressed the limitations of the current work.
paper_summary: paper_summary: The paper proposes the One-to-Multiple Progressive Style Transfer Unsupervised Domain-Adaptive (PSTUDA) framework for kidney and tumor segmentation in multi-sequence MRI, addressing inefficiencies in existing one-to-one UDA methods. PSTUDA features a multi-level style dictionary and multiple cascading style fusion modules using point-wise instance normalization to recombine content and style features progressively. Tested on private (MSKT) and public (KiTS19) datasets, PSTUDA showed 1.8% and 3.9% improvements in Dice Similarity Coefficient for kidney and tumor segmentation, respectively, and achieved significant reductions in model parameters and computational costs. The framework outperformed state-of-the-art UDA methods, enhancing both segmentation performance and training efficiency.
strength_weakness: strength_weakness: - The reported improvements in Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (HD95), while notable, are relatively modest (1.8% and 3.9%, respectively). 

- The framework's scalability to handle a larger number of target domains in real-world scenarios remains untested.

- The generalizability of this approach beyond simple kidney/tumour segmentation into multi-class intra-organ segmentation or even multi-organ segmentation remains to be seen.
questions: questions: - I would agree with the concept of the multiple cascading modules and Point-wise Instance Normalization but am surprised it doesn't benefit the overall segmentation results further. Can the authors provide their insights into why there isn't a more dramatic boost? What future work do they envision to achieve more substantial performance boosts?

-  paper demonstrates the effectiveness of PSTUDA on kidney and tumor segmentation in multi-sequence MRI. Have the authors tested the framework on other medical imaging tasks or modalities, such as CT or ultrasound? What would expected results and challenges be in other modalities?

- Despite the reduction in model parameters and FLOPs, PSTUDA still appears resource-intensive. How feasible is the deployment of PSTUDA in real-world clinical settings with limited computational infrastructure?

- Are there any plans to release code, pre-trained models, or detailed implementation guides? How can the research community best leverage PSTUDA in their own work?
limitations: limitations: Limitations and Broader Impacts are appropriately addressed.

Comments: title: Thanks for the feedback!
thank you.
Thanks for the feedback!
Thanks for the feedback!
Response to additional questions
Response to additional questions
comment: We are pleased to have addressed most of your concerns and we sincerely appreciate you raising our paper's score to 5: Borderline Accept. Your feedback has been invaluable to our work, and based on your suggestions, we will include the new supplementary experimental results in the revised version. Thank you for your support！
Most of my concerns are addressed. Hence I raised the score to 5. 
Please include the new results in the manuscript if it is accepted.
Thank you for recognizing our efforts in the rebuttal. We greatly appreciate your consideration in raising the rating of our paper. Your feedback has been invaluable to our work. Based on your suggestions, we will specifically clarify the precision used in the model in the revised version.
The authors provided bidirectional cross-modal experimental results, further demonstrating the generalization capability of the method. Additionally, they addressed each of the raised questions individually and promised to make revisions in the revised version. Therefore, I suggest changing the score to 5: Borderline Accept. 

Furthermore, there is a minor issue: I hope the authors can specify the precision used in the model, such as whether it is FP16, in the revised version.
Thank you for your positive feedback on our manuscript. We're glad our efforts to address your questions were satisfactory. We will carefully consider your suggestion to move the ablation studies into the main paper.
I appreciate the clarification on the complexities of the one-to-multiple UDA task. The trade-offs in coordinating complex mappings among multiple target domains and the resultant benefits in training time and resource costs are well-explained.

The additional validation using the new T1 MRI sequences is adequate. 

Consider moving the ablation studies into the main paper.

The simulations indicating the feasibility of deploying PSTUDA with minimal computational resources are reassuring. The specific resource requirements you provided are helpful for understanding the deployment context.

I am pleased to hear about the plans to release the code, pre-trained models, and implementation guidelines upon acceptance.

Overall, your rebuttal has adequately met my concerns. I maintain my score as it is.
As we mentioned in the main rebuttal, we include the answers to the additional questions of Reviewer **tgyp**. We hope that this helps to address all remaining concerns and we thank again for taking the time to review our work.

**Q1. Overhead and warm-up for multi-level style dictionary.**

In our experiments, the multi-level style dictionary essentially consists of a set of learnable tensors. Content is accessed through indexing, and as the style vectors are independent of each other, this does not result in any additional overhead.

Our method primarily focuses on adapting a source domain to multiple target domains simultaneously. All domains are fixed and visible during training, so there are no encounters with new unknown domains. Additionally, the multi-level style dictionary is randomly initialized at the beginning of training and does not require a warm-up phase.

**Q2. Size of Multi-level Style Dictionary.**

In our experiments, the multi-level style dictionary is directly created as a set of learnable tensors. Specifically, its dimensions are (4, 6, 4096), where 4 represents the number of domains, 6 represents the number of style fusion layers (four layers in the style fusion module and two layers in the decoder), and 4,096 represents the depth of the style dictionary. Therefore, this tensor contains a total of 4×6×4096=98,304 parameters, which converts to approximately 0.10M.

**Q3. Generator and Discriminator Architecture.**

* **Generator:** PSTUDA uses the CycleGAN generator. The difference lies in the replacement of normalization operations in the style fusion module and decoder from AdaIN to our proposed PIN. Additionally, we employ a progressive style injection method layer by layer to achieve image translation.

* **Discriminator**: PSTUDA’s discriminator architecture is based on the multi-scale discriminator from MUNIT. The difference is that we replace the Conv2dBlock in MUNIT with ResBlk, which has a residual structure. Additionally, the multi-scale output sizes are set to 1/16, 1/32, 1/64, and 1/128 of the original image size. Our motivation for using a multi-scale discriminator is that the generator, integrated with PIN, will have enhanced generative performance. Therefore, the discriminator also needs to be more powerful to match the generator, enabling better adversarial training and effectively leveraging the generator’s capabilities.

[1] Kavur, A. E., et al. CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation. MedIA 2021.

[2] Landman, B., et al. Multi-Atlas Labeling Beyond the Cranial Vault. 2017.

[3] Huang, X., et al. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. ICCV 2017.

[4] Nam, H., et al. Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks. NeurIPS 2018
As we mentioned in the main rebuttal, we include the answers to the additional questions of Reviewer **yJwR**. We hope that this helps to address all remaining concerns and we thank again for taking the time to review our work.

**W3. Ablation Study on key hyperparameters affecting model performance.**

 Our main contributions lie in the multi-level style dictionary and the style fusion module. These two modules have the following hyperparameters: the depth of the style dictionary, the number of levels in the style dictionary, and the number of style fusion modules. Note that the number of levels in the style dictionary is equal to the number of style fusion modules.

To investigate the sensitivity of model performance to various hyperparameters, according to your suggestions, we conducted extensive ablation studies on the depth of the multi-level style dictionary (MSD) and the number of style fusion modules (SFM). The results, presented in the table below, indicate that under the original settings (dictionary depth of 4,096, module number of 4), PSTUDA performs optimally in most experiments. Notably, when we extended our dictionary depth to 16,384, there was an improvement in segmentation performance on T2W sequence.

| **MSD Depth** |           | **T1c→FS T2W** |           |           | **T1c→T2W** |           |           | **T1c→DWI** |           |
| :-----------: | :-------: | :------------: | :-------: | :-------: | :---------: | :-------: | :-------: | :---------: | :-------: |
|               |  Kidney   |     Tumor      |   Avg.    |  Kidney   |    Tumor    |   Avg.    |  Kidney   |    Tumor    |   Avg.    |
|      256      |   84.85   |     66.34      |   75.60   |   70.55   |    43.85    |   57.20   |   83.15   |    63.52    |   73.34   |
|     1024      |   85.84   |     69.68      |   77.76   |   70.06   |    42.76    |   56.41   |   83.99   |    64.41    |   74.20   |
|     4096      | **86.30** |   **76.36**    | **81.33** |   77.26   |    53.77    |   65.52   | **86.99** |  **74.23**  | **80.61** |
|     16384     |   84.63   |     66.74      |   75.69   | **77.63** |  **57.49**  | **67.56** |   83.31   |    68.78    |   76.05   |

| **SFM Number** |           | **T1c→FS T2W** |           |           | **T1c→T2W** |           |           | **T1c→DWI** |           |
| :------------: | :-------: | :------------: | :-------: | :-------: | :---------: | :-------: | :-------: | :---------: | :-------: |
|                |  Kidney   |     Tumor      |   Avg.    |  Kidney   |    Tumor    |   Avg.    |  Kidney   |    Tumor    |   Avg.    |
|       2        |   85.65   |     71.77      |   78.71   |   73.94   |    44.51    |   59.23   |   83.89   |    61.31    |   72.60   |
|       4        | **86.30** |   **76.36**    | **81.33** | **77.26** |  **53.77**  | **65.52** | **86.99** |  **74.23**  | **80.61** |
|       6        |   84.49   |     67.94      |   76.22   |   74.97   |    50.57    |   62.77   |   81.20   |    60.44    |   70.82   |
|       8        |   84.09   |     63.86      |   73.98   |   60.06   |    30.43    |   45.25   |   83.75   |    60.60    |   72.18   |

[1] Huang, X., et al. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. ICCV 2017.

[2] Nam, H., et al. Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks. NeurIPS 2018.