Hyperparameter-Free Medical Image Synthesis for Sharing Data and Improving Site-Specific Segmentation

Authors: alexander  chebykin
peter a n bosman
tanja  alderliesten
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 23:48:11

Abstract: Sharing synthetic medical images is a promising alternative to sharing real images that can improve patient privacy and data security. To get good results, existing methods for medical image synthesis must be manually adjusted when they are applied to unseen data. To remove this manual burden, we introduce a Hyperparameter-Free distributed learning method for automatic medical image Synthesis, Sharing, and Segmentation called HyFree-S3. For three diverse segmentation settings (pelvic MRIs, lung X-rays, polyp photos), the use of HyFree-S3 results in improved performance over training only with site-specific data (in the majority of cases). The hyperparameter-free nature of the method should make data synthesis and sharing easier, potentially leading to an increase in the quantity of available data and consequently the quality of the models trained that may ultimately be applied in the clinic. Our code is available at https://github.com/AwesomeLemon/HyFree-S3

Main paper: 
 introduction: 
 Deep learning methods can be beneficial for medical applications, but often suffer from limited data availability (Bowles et al., 2018) . Generating and sharing synthetic datasets was suggested as a viable solution (Dube and Gallagher, 2014) .Many approaches can synthesize medical images (Bowles et al., 2018; Yi et al., 2019) , fewer jointly produce segmentation maps (Guibas et al., 2018; Han et al., 2023 ) (which would be required for training down-stream segmentation models) and, to the best of our knowledge, not a single method exists that could be applied to a new task without manually adjusting hyperparameters of training or data preprocessing.The benefits of a hyperparameter-free adaptive method are self-evident. Such a method was realized for segmentation tasks by nnU-Net (Isensee et al., 2021) , a method of automatically adjusting architecture and hyperparameters of a U-Net (Ronneberger et al., 2015) that showed excellent and robust performance in tens of challenges (Isensee et al., 2021) .Automatic adaptation of a high-quality underlying model and training pipeline is a general idea that could be extended to other tasks, such as medical image synthesis. A Generative Adversarial Network (GAN) called StyleGAN2 (Karras et al., 2020) demonstrated good results in medical image synthesis (Woodland et al., 2022) . However, it currently does not automatically adapt to the image dimensions or the dataset size.In this paper, we introduce an automatically adjustable StyleGAN2 setup and integrate it with nnU-Net to create a Hyperparameter-Free medical image Synthesis, Sharing, and Segmentation method called HyFree-S3. We construct it as a distributed learning method where each site (e.g., a hospital) can automatically and asynchronously create a synthetic dataset and share it. A segmentation model can be automatically trained on the merged synthetic data and distributed back to the sites to be further automatically fine-tuned for improved performance on local data (see Figure 1 ).This approach to distributed learning has practical advantages of requiring minimal coordination between sites and of reduced privacy risk thanks to not sharing real data, models trained with it, or their gradients (which is a potential source of data leakage in federated learning (Zhu et al., 2019) ). An important concern is whether synthetic data includes memorized real data, which we address with a quantitative and qualitative investigation, as well as a technique for ensuring that synthetic data is not too similar to the real data.We evaluate our method in three segmentation settings (pelvic MRIs, lung X-rays, polyp photos) to test its generality, the impact of synthetic data sharing, and the difference in performance compared to the realistic baseline of using only local data and the strong baseline of having central access to all the real data.In this paper, we only consider 2D models: compared to 3D models, they have lower computational requirements and need less data to be trained (which is important for the data sharing setting where some sites could have small datasets). In the future, our approach could be extended to 3D models for improved segmentation performance in settings where there is enough data and computational resources.The contributions of this paper are as follows:• We propose HyFree-S3, a hyperparameter-free distributed learning method integrating image synthesis, data sharing, and segmentation. • Towards that goal, we introduce a hyperparameter-free StyleGAN2 setup that can adapt to various image dimensions and dataset sizes. • The segmentation quality of HyFree-S3 is evaluated in three settings. Furthermore, its scaling behavior and ability to avoid data memorization is investigated. 
sharing synthetic medical image data: 
 Synthesizing medical data for sharing purposes so as to avoid privacy issues is an established idea in the literature (Goncalves et al., 2020) . Medical image data can be synthesized by GANs (Bowles et al., 2018; Yi et al., 2019; Sun et al., 2022) , but for supervised learning to be possible, an annotation needs to be associated with a synthetic image. While conditioning GANs on an image class is straightforward when generating data for classification tasks (Hu et al., 2018) , conditioning on segmentation maps (Chang et al., 2023) to generate data for segmentation tasks requires the additional complexity of sharing segmentations to condition on (e.g., by training a separate GAN for them (Guibas et al., 2018; Han et al., 2023) ). Thambawita et al. (2022) propose a GAN for joint unconditional generation of polyp images and segmentations that improves performance but requires a separate GAN to be trained for each input image, likely impeding scaling to larger datasets. Federated learning (Konečný et al., 2016) can be applied to medical imaging data (Rieke et al., 2020; Adnan et al., 2022) where it allows the global model to be trained on diverse data from multiple hospitals (Ng et al., 2021) without sharing local data. Federated learning algorithms for training a GAN (Rasouli et al., 2020) can be applied to medical data (Chang et al., 2023) but they incur privacy costs because real data could be reconstructed from the gradients passed between sites (Zhu et al., 2019) . A common solution (Chang et al., 2020 (Chang et al., , 2023;; Wang et al., 2023) is to train only the generator globally while the discriminators are trained per-site using the local data and the synthetic data from the global generator.Our method differs from the existing distributed learning techniques in three ways. Firstly, it is asynchronous and does not require simultaneous online access to sites. Secondly, the generated data is filtered to not contain memorized data (in contrast to sharing a black-box generative neural network with potentially undiscovered vulnerabilities). Finally, HyFree-S3 is adaptable and hyperparameter-free, thus making it potentially easier to use. 
overview of the method: 
 We assume that N sites (such as medical centers) have a goal of solving a segmentation task. Each site has a local dataset that cannot be straightforwardly shared due to privacy or security concerns. The datasets may differ in sizes and image characteristics per-site.Figure 1 shows the flow of data and models in HyFree-S3. Firstly, each site runs hyperparameter-free methods to create a generative model and a segmentation model. A generative model (Section 3.3) is used to create data without segmentations, which are then segmented by the segmentation model (Section 3.2) to create a complete synthetic dataset for sharing. The reasoning behind using two separate models is given in Section 3.4.Next, the synthetic datasets from all sites are merged in a central location, and a general segmentation model is trained using all the synthetic data. This general segmentation model is transferred back to the sites, and is further automatically fine-tuned at each one. The resulting models benefit from the general pretraining but are specialized for each site.nnU-Net is a robust medical image segmentation method that was shown to perform excellently in a wide variety of competitions and benchmarks (Isensee et al., 2021) . nnU-Net can adapt to diverse datasets without hyperparameter tuning thanks to heuristics for adjusting the underlying U-Net architecture (Ronneberger et al., 2015) and the training procedure. The spatial dimensions of the data influence the input size of the model, depths of the encoder (decoder), downsampling (upsampling) strides, and convolution kernel sizes.To adjust the hyperparameters to the fine-tuning setting not considered by Isensee et al. (2021) , we add linear learning rate warm-up for the first 10% of training epochs (Mosbach et al., 2020) and do not otherwise change the default hyperparameters.StyleGAN2 (Karras et al., 2020) is a powerful generative model that by default generates square images of a resolution of a power of two. However, medical images come in a variety of resolutions. Ideally, synthetic images of the appropriate resolution should be generated.For segmentation tasks, nnU-Net adapts the structure of the U-Net to the resolution. We noticed a similarity between the structures of the generator/discriminator of a GAN and those of the decoder/encoder of a U-Net. Both the GAN generator and the U-Net decoder gradually upscale a low-resolution many-channel latent representation of an image towards a high-resolution few-channel output. The architectures need to strike the correct balance between the speeds of increasing the resolution and decreasing the number of channels. As such, a network that strikes the correct balance in one setting, seems likely to do so in the other. Analogously, the discriminator and the encoder gradually downscale a high-resolution few-channel input towards a low-resolution many-channel representation.Therefore, it appears natural to reuse the hyperparameters of the encoder and the decoder automatically determined by nnU-Net (depths of the networks, convolution strides, and kernel sizes) for the discriminator and the generator of a GAN. This will allow it to create non-square non-power-of-two-sized images of the exact size determined by nnU-Net.Next, the number of training steps, n steps , needs to be set automatically. In StyleGAN2, n steps is defined in thousands of real images processed during training. As the dataset size increases, so should n steps (to allow the GAN to learn from a larger amount of data). Setting n steps to the number of images in the dataset ensures good image quality across different dataset scales, keeps training times short, and prevents memorization (see Section 4.3.3).The number of images to be generated, n gen , also needs to be determined. While generating images with GANs requires little compute and time, generating increasingly large numbers of samples leads to diminishing returns (Ravuri and Vinyals, 2019) . We also need to consider the proportions of synthetic data coming from different sites used in training the general segmentation model: generative models trained on larger datasets should contribute more than those trained on smaller datasets, as they likely have higher quality. For these reasons, n gen is set to ten times the dataset size for each dataset.We use the augmentations setup of Zhao et al. (2020) that was shown to improve performance and help avoid overfitting with datasets as small as 100 images.Segmenting with a separate model was a deliberate design choice and constraint. While it is possible to train a generative model to output segmentations as well as images, this would lead to segmentations influencing the images. This is undesirable for medical data sharing, as the references in many segmentation scenarios vary due to the protocol and observer variation. Letting these variations influence the generated images should be avoided: then the images themselves can still contribute to the improvement of the models (e.g., during unsupervised pretraining) even if segmentations need to be discarded or redone. Additionally, if no annotations are available, HyFree-S3 can still be utilized to share images (unlike methods that condition on segmentations).Synthetic data should be similar enough to the real data for the models trained on one to transfer to the other, and dissimilar enough for the privacy concerns to be alleviated. Generative models are capable of memorizing their training data and outputting it as "synthetic" samples (Feng et al., 2021) . Memorization is difficult to determine because the reproduction typically includes some variation or noise. It is not obvious where the threshold between the presence and the absence of memorization should be.We propose automatically determining this threshold based on the real data itself as follows: firstly, the patients are randomly split into two subsets. For each image in the first subset, its dissimilarity with each image in the second subset is computed using the L2 distance between their OpenCLIP embeddings (Ilharco et al., 2021) . The minimal dissimilarity (i.e., the distance to the nearest neighbour) is stored for each image. The threshold is then defined as the p-th percentile of these dissimilarities.After the synthetic dataset is generated, the dissimilarity of each synthetic image to all real images is similarly computed. If the distance of a synthetic image to its nearest realimage neighbor is below the determined threshold, the image is declared to be memorized and is discarded. For p = 0 (the minimum of dissimilarities), this procedure ensures that any synthetic image is only as similar to a real image as one unrelated real image to another; we set p = 5 to guard against outliers. The procedure relies on the quality of the embedding model that can only be demonstrated empirically (Cherti et al., 2023) . 
experiment setup: 
 In our experiments, we emulate a distributed setting with N sites. As per Section 3.1, for each site S i , GAN-S i and U-Net-real-S i are trained and used to generate a synthetic dataset. U-Net-syn-all is trained on the merged datasets and fine-tuned at each site to get U-Net-syn-real-S i . U-Net-real-all is trained on merged real data as a baseline. All the experiments are performed via five-fold cross-validation (3 folds for training, 1 fold for validation and test each). The mean and the standard deviation of the Dice Score (DS) and the 95th percentile of the Hausdorff Distance (HD95) across the folds are reported. All the evaluations were performed on real data. The results of statistical testing are given in Appendix A. Appendix D contains comparisons with federated learning baselines. The ablations of pretraining with single-site synthetic data and of using the standard StyleGAN2 architecture are reported in Appendices F, G. Figure 2 shows that in the experiments with Cervix, DS and HD95 metrics improve in most cases. The performance on site A is approximately the same across the real-A, synreal-A, real-all settings, showing that adding data from site B is not very helpful even if it is real data. This is likely due to the large number and uniformity of patients in A itself. Nonetheless, the performance in the syn-real-A setting on site B is improved compared to real-A (by 3.3 DS and 4.3 HD95 on average), showing that pretraining on merged synthetic data improves the robustness of the model to data shifts. For site B, the pretrained and fine-tuned model (syn-real-B ) outperforms its counterpart trained exclusively on the local data (real-B ) when tested on both A (by 6.3 DS and 4.3 HD95 on average) and B (by 2.2 DS and 2.5 HD95 on average). The full results for all datasets are given in Appendix B.It can be seen in Figure 3 that switching from the model trained only on local data (real ) to the one pretrained on the merged synthetic data and fine-tuned on local data (syn-real ) leads to improvements in the Lung setting of 0.8 DS on average. For Polyp, the improvement for the target site is minor (0.5 DS on average), but the improvement in robustness to data shifts, as measured by the performance on the other site, is large (on average, 2.7 DS for A and 13.4 DS for B).While training on real data centrally (real-all ) gives the best results overall, our method comes close (the largest difference is 2.0 DS in Polyp-B) without requiring real data sharing. The scaling behavior of HyFree-S3 is investigated in the Lung setting, with the data split into 8 sites. The general segmentation model is pretrained with the synthetic data from 2, 4, or 8 sites, and then fine-tuned at each site. The average difference in DS between real (training with local data only) and syn-real, shown in Figure 4 , becomes larger as the number of site grows, showing that the method is scalable and enables larger improvements when more sites join.Per Section 3.5, we use OpenCLIP embeddings to compare either subsets of real images, or real images and synthetic images. As a sanity check, we established that an image present in two sets will be its own nearest neighbor, and that a mirrored image will most often be the nearest neighbor of the original image (in 97.9% of cases for Lung).Figure 5 (top right) shows a histogram of distances to the nearest neighbor when comparing two subsets of real images (from a Lung experiment), as well as their 5 th percentile that will be used as a threshold to filter synthetic images. The histogram of distances between real and synthetic images in Figure 5 (bottom right) has a similar shape but shifted to the right. As a result, all synthetic images have a distance above the threshold, meaning that while synthetic images are generally similar to real images, they are not too similar.We visualize adversarially chosen real and synthetic images in Figure 5 (left). We select two synthetic images (column 2) with the smallest nearest neighbor distances to some real images (column 1), and the synthetic images that have the second-smallest distances to these real images (column 3). The images are broadly similar but clearly distinct and do not demonstrate memorization (it should be noted that our memorization analysis relies on OpenCLIP embeddings, see Appendix E for further discussion of memorization).For Lung and Polyp, only 6 synthetic images (out of 3 × 10 5 ) are discarded as too similar to real images. For Cervix, 1,180 images (out of 2.6 × 10 5 ) are discarded, some of which look very similar to their real nearest neighbours but none are exact duplicates, which is consistent with the nature of 3D data where nearby slices are similar. See Appendix C for visual comparisons between real and synthetic images. Only 0.2% of synthetic images being discarded shows that our GANs typically do not memorize images but the proposed filtering scheme could nonetheless help to avoid sharing potentially memorized data.  
discussion & conclusion: 
 We have introduced HyFree-S3, a method for synthesizing medical images for sharing and segmentation that does not require adjusting hyperparameters, as it relies on our hyperparameter-free setup of StyleGAN2 for image generation and similarly hyperparameterfree nnU-Net (Isensee et al., 2021) for segmentation.The method is asynchronous and does not expect different sites to share infrastructure or to be online simultaneously. HyFree-S3 was shown to come close to the performance of training with centralized real data and to improve upon training only with local data, while not requiring sharing real patient data. The synthetic dataset produced by HyFree-S3 could be reused for training future models without going back to the sites where the real data is stored. We additionally proposed a memorization evaluation technique for ensuring that synthetic data is sufficiently dissimilar from real data.However, our method also has a number of limitations. Firstly, as models for data sharing are trained on each site independently, large enough datasets have to be present there (especially for training a GAN). Secondly, we generate images and annotations separately for reasons described in Section 3.4, which nonetheless means that the annotations produced by an independent model will not perfectly correspond to the images, introducing noise to synthetic data. Finally, our method as presented cannot be applied to all possible datasets, as it relies on nnU-Net for preprocessing and segmentation: nnU-Net can handle a variety of inputs but not, e.g., high-resolution histopathology images. However, extending nnU-Net with an appropriate data preprocessing procedure would make HyFree-S3 applicable too.Despite some limitations, adaptable methods that do not require task-specific tuning are still valuable for the translation of deep learning models into the real world, where there may not be time or expertise for careful manual adaptation to each task. Since the key contribution of our method is its automatic and hyperparameter-free nature, we run the baselines in a similar setting of no hyperparameter tuning. For FedAvg, we used 1000 communication rounds with 1 epoch of local training in between since it was the best setting in (McMahan et al., 2017) . For DSL, we followed the paper (Chang et al., 2023) and the official code base. For hyperparameters that differed across the three datasets in DSL, we used the median values: λ L1 : 300, 150, 100 → 150; batch size: 6, 6, 3 → 6).The results of the experiments are reported in Tables 6, 7, 8. FedAvg on average has 5.3 worse Dice Score (DS) than HyFree-S3 in the i.i.d. setting of Lung, with the gap widening further in the non-i.i.d settings: 10.7 DS for Cervix, 29.2 DS for Polyp. DSL performs substantially worse in all settings (on average: Lung: 32.8 DS worse, Cervix: 35.1 DS worse, Polyp: 36.9 DS worse). The poor performance of DSL was unexpected, given the excellent results in (Chang et al., 2023) . We attribute this to untuned hyperparameters. We have checked that it learns to generate relatively realistic images (see Figure 8 ), on which the U-Net that it trains performs well, however it generalizes to real test images poorly.We additionally did minimal manual hyperparameter tuning of DSL in the Polyp setting and were able to improve its performance by 9.2 DS on average ("DSL (lightly tuned)" in Table 8 ; we only changed the hyperparameters of the U-Net: we switched the optimizer to AdamW with default hyperparameters, switched step-wise learning rate schedule to cosine annealing, added random color jitter and random rotation augmentations).While the performance of FedAvg and DSL could potentially be improved further via hyperparameter tuning, their default performance is subpar, highlighting the benefit of automatic methods such as HyFree-S3, the hyperparameter-free nature of which is its key benefit. Appendix E. Further discussion of memorizationWe have been communicating with a data protection officer about the perspectives of synthetic data since this project started. Based on preliminary discussions, our method could greatly simplify data sharing between institutes and decrease privacy risks. However, currently, there are no official, internationally accepted, guidelines as to what constitutes memorization (of imaging data).Therefore, it is difficult to say if our approach is "clinically acceptable", given lack of prior work on what that entails. While we are confident in our analysis, it could be made more robust by using several dissimilar embedding models, or by increasing the threshold. Still, our method relies on empirical performance of neural networks (as noted in Section 3.5), and therefore no mathematical guarantees can be given that no memorization occurs. Nonetheless, we believe that empirical evaluation similar to ours could be enough for clinical acceptance, once the guidelines are determined.To further investigate the memorization phenomenon, we use an alternative method for finding memorized images from a concurrent work (Dar et al., 2024) , where it was successfully used to find medical images memorized by a diffusion model. The method is similar to the approach we used in that it relies on embeddings of images via a neural net, with the two differences from our work being the model (the authors trained their own embedding model on the target dataset) and the similarity measure (the authors used correlation). Distance (FID) (Heusel et al., 2017) is calculated between them and the real images. FID is an established metric of GAN quality, lower values are better. We compare the FID of the StyleGAN2-256 × 256 to our StyleGAN2-388 × 320, and find the latter to achieve better results (102.4 ± 4.0 FID vs 88.0 ± 3.2 FID). This confirms that the architecture is reasonable, as it was able to make use of the increased resolution of its inputs to achieve higher generation quality. 
 
 Appendix:  


Decision: Most of the reviewers’ critiques were addressed in the authors’ feedback, including a comparison with other federated learning methods. Reviewers acknowledged that the paper has been revised sufficiently. Although a question still remained by a reviewer (using other image generation methods rather than the proposed method), the proposed idea seems to be still worth discussing in MIDL.

Reviews: review: paper_summary: paper_summary: This paper presents a hyperparameter-free sitribution learning method for medical image synthesis, sharing, and segmentation. The method consists of a variant of StyelGAN2, motivated by nnU-Net, for hyperparameter-free data synthesis, a nnU-Net based segmentation model on merged synthetic data, and a strategy to recognize generated samples that may be a memorization of real samples. The experiment results show that the sharing of these synthetic data improves the segmentation performance on the local sites and approaches that can be obtained by training on combined real data.
main_review: main_review: The authors have addressed my previous main comments in terms of adding comparative analyses. I think the paper will have good discussion value for MIDL. I will thus stay with my original rating and recommend the acceptance of the paper.
strength_weakness: strength_weakness: While the experimental results demonstrated the benefit of the proposed work to merge synthetic data for training, it’d be good to see comparisons to other image synthesis and federated learning works to provide empirical evidence for the limitations discussed in the related works section.
questions: questions: Adding some comparative analyses will largely strengthen the work.
paper_summary: paper_summary: The authors propose a distributed learning approach that allows for data synthesis, even in data sensitive scenarios, with an application in image segmentations. They propose the training of an image synthesis model per site, which can be combined to generate synthetic data from all sites, containing site-specific but not sensitive data. The proposed method outperforms models trained on data from a single site.
main_review: main_review: I would like to thank the authors for their thorough responses. I believe that my main concerns, and the concerns of the other reviewers have been adequately addressed, and taking all the review comments and responses into consideration, I am changing my final rating.
strength_weakness: strength_weakness: The authors show that their method significantly outperforms models trained on a single site. However, the real target would be to show that training on synthetic data is not significantly worse than training on all the real data available.
questions: questions: -  Using the pre-processing steps of nnUNet is certainly advantageous, however selecting the hyperparameters of the encoder and the decoder is more questionable. Although the feature space sizes of the encoder/decoder of a UNet and the generator/discriminator of a GAN are similar, their objectives are entirely different. The reviewer thinks it does not "appear natural to reuse the hyperparameters" determined by the nnUNet in the UNet architecture. Did the authors perform experiments to show that this is a reasonable decision?
- The results claim that "the images are broadly similar but clearly distinct and do not demonstrate memorization". I think it's important to note that this is according to the authors' definition of memorization based on OpenCLIP embeddings. What do the authors think about the clinical implications of this method? Do they believe that their results are "clinically acceptable"? Would it be allowed to use this threshold to distribute synthetic data generated by such a model? Or is there still a chance of memorization? I believe the approach is very interesting, however it should be evaluated further to be trustworthy.
paper_summary: paper_summary: This paper introduces an automatically adjustable StyleGAN2 setup and integrates it with nnU-Net to create a Hyperparameter-Free medical image Synthesis, Sharing, and Segmentation method called HyFree-S3. The method is evaluated on three segmentation settings.
main_review: main_review: The rebuttal solved most of my concerns, although I am still not so convinced by the problem setting in practice. 
As for generating more samples for one site: I think the mentioned diversity limitation can be easily solved using advanced generative models. The domain shift of scanners can also be readily simulated, for example, by using some techniques in domain generalization like exchanging the style using FFT. Thus, It is hard to discern if the issue has been resolved because the added empirical results are possibly dependent on the limitation of the used generative methods. Or for GAN-based methods, you can use GAN to generate more samples and followed by a style transformation used in domain generalization for one site data.
strength_weakness: strength_weakness: 1.	Please discuss the federated learning in this paper, and compare their advantages and disadvantages. Because your setting is like a synthetic centralized learning to protect privacy, comparison with decentralized methods is also very important to position this setting.
2.	Motivation is still not clear. Why not just use one site to synthesize a large scale and diverse data pool, but use complicated multi-site collaboration to achieve the goals? So far, I am not convinced by the problem setting.
3.	Experiments should be improved: like question 2, use one site but generate more samples for training -> discard the collaborative training; federated learning should be compared.
questions: questions: See above

Comments: title: Response to reviewer GiX9
Response to the authors
Official Comment by Authors (Part 2)
Official Comment by Authors (Part 1)
Official Comment by Authors (Part 3)
Official Comment by Authors (Part 2)
Official Comment by Authors (Part 1)
comment: We are glad that no further unclarities remain, and we thank the reviewer for helping us improve the paper.
I would like to thank the authors for their thorough response to my questions, and the questions of the other reviewers. I hope a conversation with Reviewer EtyB will happen during the rebuttal phase. I believe that their main concerns have all been adequately addresssed as well, therefore I am changing my rating appropriately.
We thank all the reviewers for their insightful comments. We have addressed each point raised by the reviewers in the individual responses below, here we provide a summary of the main changes in the updated version of the paper:
1. Added a comparison to two federated learning baselines (Federated Averaging, Distributed Synthetic Learning), see Appendix D.
2. Extended the discussion of memorization, see Appendix E.
3. Added a comparison of pretraining on single-site synthetic data to pretraining on multi-site synthetic data, see Appendix F.
4. Added a comparison with the standard StyleGAN2 that does not use the adaptive architecture parameters, see Appendix G.
_(this is the continuation of Part 1 of our response)_

> “Federated learning should be compared.”

To compare our approach to federated learning, we run two federated learning baselines: Federated Averaging (FedAvg) [1] (in combination with nnU-Net) and Distributed Synthetic Learning (DSL) [2], a state-of-the-art approach to training a GAN in a federated manner to be used for generating synthetic data and training a U-Net. Since the key contribution of our method is its automatic and hyperparameter-free nature, we run the baselines in a similar setting of no hyperparameter tuning (for FedAvg, we used 1000 communication rounds with 1 epoch of local training in between since it was the best setting in [1]; for DSL, we followed the paper and the official code base (https://github.com/tommy-qichang/DSL_All_Code), using the median values for hyperparameters that differed across the three datasets in DSL: λ-L1: 300, 150, 100 → 150; batch size: 6, 6, 3 → 6). 

The results of the experiments are reported in the Appendix D. FedAvg on average has 5.3 worse Dice Score (DS) (on the scale 0-100) than our HyFree-S3 in the i.i.d. setting of Lung, with the gap widening further in the non-i.i.d settings: 10.7 DS for Cervix, 29.2 DS for Polyp. DSL performs substantially worse in all settings (on average over 5 folds: Lung: 32.8 DS worse, Cervix: 35.1 DS worse, Polyp: 36.9 DS worse). The poor performance of DSL was unexpected, given the excellent results reported in [2]. We attribute this to untuned hyperparameters. We have checked that it learns to generate relatively realistic images (see Figure 8), on which the U-Net that it trains performs well, however it generalizes to real test images poorly. We additionally did minimal manual hyperparameter tuning of DSL in the Polyp setting and were able to improve its performance by 9.2 DS on average (see Appendix D for details).

While the performance of FedAvg and DSL could potentially be improved further via hyperparameter tuning, their default performance is subpar, highlighting the benefit of automatic methods such as HyFree-S3, the hyperparameter-free nature of which is its key benefit.

We updated our code to include the FedAvg implementation: https://github.com/AwesomeLemon/hyfree-s3-midl-review. For the DSL experiments, see our fork of the official code base: https://github.com/AwesomeLemon/DSL_All_Code.

We believe that we addressed all the concerns of the reviewer. If any unclarities remain, we would be happy to clear them up.

[1] McMahan, B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. y. (2017). Communication-Efficient Learning of Deep Networks from Decentralized Data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 1273–1282. https://proceedings.mlr.press/v54/mcmahan17a.html

[2] Chang, Q., Yan, Z., Zhou, M., Qu, H., He, X., Zhang, H., Baskaran, L., Al’Aref, S., Li, H., Zhang, S., & Metaxas, D. N. (2023). Mining Multi-Center Heterogeneous Medical Data with Distributed Synthetic Learning. Nature Communications, 14(1). https://doi.org/10.1038/s41467-023-40687-y
We thank the reviewer for their valuable feedback.

> “1. Please discuss the federated learning in this paper, and compare their advantages and disadvantages. Because your setting is like a synthetic centralized learning to protect privacy, comparison with decentralized methods is also very important to position this setting.”

We agree that federated learning is a relevant alternative to our approach, therefore the paper already includes a discussion of federated learning in the related work (section 2.2), as well as a comparison of the properties of our approach to prior work. The limitations of our method are discussed in the conclusion (section 5). If our discussion omits important details, we would appreciate them being pointed out so that we could include them.

> “2. Motivation is still not clear. Why not just use one site to synthesize a large scale and diverse data pool, but use complicated multi-site collaboration to achieve the goals? So far, I am not convinced by the problem setting.”

The motivation for using data from more than one site is that a single site has a limited amount of data, covering only a part of the true data distribution. While generative models could learn to generate an infinite amount of data similar to their training data, they will not generate data outside their training distribution. For example, a GAN trained on data collected with scanner X will never produce images similar to those collected with scanner Y (or: a GAN trained on the images of cats will never produce an image of a dog). At the same time, deep learning models benefit from an increased amount of diverse data. Therefore, training segmentation models on data from multiple sites should improve their performance and generality.

> “3. Experiments should be improved: like question 2, use one site but generate more samples for training -> discard the collaborative training;”

To compare generating more training samples from one site against generating samples from multiple sites, we run an additional experiment and report it in Appendix F. Specifically, we compare our default setting (generating 10 times the number of real images at each site and pulling them together) to the setting of generating 20 times the number of real images at one site. In both settings, a U-Net is pretrained with the synthetic images and fine-tuned with the real local data. The experiments are performed for two sites with the Cervix data (5 folds). The networks pretrained with the local synthetic data (syn-local-real) achieve on average 2.7 Dice Score (DS) worse performance than the networks pretrained on pooled synthetic data (syn-real). This serves as an empirical confirmation of our reasoning behind using multi-site synthetic data as outlined above.

_(the response is continued in Part 2 due to character limit)_
_(this is the continuation of Part 2 of our response)_



To further investigate the memorization phenomenon, we use an alternative method for finding memorized images from a concurrent work [4], where it was successfully used to find medical images memorized by a diffusion model. The method is similar to the approach we used in that it relies on embeddings of images via a neural net, with the two differences from our work being the model (the authors trained their own embedding model on the target dataset) and the similarity measure (the authors used correlation).

We apply this approach in the Lung setting (see Appendix E). Whereas our method flagged close to zero samples as memorized, this approach flags ~12%. However, visual inspection of the synthetic samples closest to some real ones (Figure 9) indicates that the synthetic samples are not duplicates of the real ones (based on our judgement). While similar, they differ in their details, and do not satisfy the properties on which the definition of memorization from [4] is based: they are not variants of the original image derived via rotation, flipping, or contrast adjustment.

Nonetheless, memorization is difficult to define, and perhaps the eventual guidelines would enforce stricter definitions of memorization that would include these samples. If so, such synthetic outputs could be removed by using an appropriate embedding model within our method. HyFree-S3 is agnostic to the duplicates removal technique used within, better techniques can be substituted when they are developed. As memorization is a complicated topic that is not the focus of our paper, we leave deeper investigation to future research.

We believe that we addressed all the questions of the reviewer. If any remain, we would be glad to answer them.

[1] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems, 30. https://papers.nips.cc/paper_files/paper/2017/hash/8a1d694707eb0fefe65871369074926d-Abstract.html

[2] Pinaya, W. H., Tudosiu, P. D., Dafflon, J., Da Costa, P. F., Fernandez, V., Nachev, P., ... & Cardoso, M. J. (2022, September). Brain imaging generation with latent diffusion models. In MICCAI Workshop on Deep Generative Models (pp. 117-126). Cham: Springer Nature Switzerland.

[3] Hamamci, I. E., Er, S., Simsar, E., Tezcan, A., Simsek, A. G., Almas, F., ... & Menze, B. (2023). GenerateCT: Text-Guided 3D chest CT generation. arXiv preprint arXiv:2305.16037.

[4] Dar, S. U. H., Seyfarth, M., Kahmann, J., Ayx, I., Papavassiliu, T., Schoenberg, S. O., & Engelhardt, S. (2024). Unconditional Latent Diffusion Models Memorize Patient Imaging Data (arXiv:2402.01054). arXiv. https://doi.org/10.48550/arXiv.2402.01054
_(this is the continuation of Part 1 of our response)_

To **experimentally confirm** that transferring the nnU-Net architectural parameters to StyleGAN2 is reasonable, we compare our GANs to the standard StyleGAN2 trained with square images of a resolution of a power of two (see Appendix G). The experiment is performed with the Polyp-B data, for which the nnU-Net determined resolution (388\*320) is the farthest from a square where the dimensions of the sides are a power of two (note that for such images our StyleGAN2 architecture would be exactly the same as the standard one). The images are resized such that the sides are the closest power of two (256\*256), and the standard StyleGAN2 is trained. Then, synthetic images are generated and resized to 388\*320. Afterwards, the Frechet Inception Distance (FID) [1] is calculated between them and the real images. FID is an established metric of GAN quality, lower values are better. 

We compare the FID of the StyleGAN2-256x256 to our StyleGAN2-388x320, and find the latter to achieve better results (102.4 ± 4.0 FID vs 88.0 ± 3.2 FID). This confirms that the architecture is reasonable, as it was able to make use of the increased resolution of its inputs to achieve higher generation quality.

Additional evidence that the GAN architectures are reasonable comes from our main experiments, where pretraining on the synthetic data produced by the GANs leads to improved performance on real data (this shows that the GAN architectures were successfully trained to capture the essence of the real data, which is their purpose).

> The results claim that "the images are broadly similar but clearly distinct and do not demonstrate memorization". I think it's important to note that this is according to the authors' definition of memorization based on OpenCLIP embeddings.

To clarify, in the context of that paragraph, we meant that no memorization is observed upon visual inspection, but given that the images were selected for visualization based on OpenCLIP embeddings, the reviewer is correct, we added the appropriate note to section 4.3.3.

> What do the authors think about the clinical implications of this method? Do they believe that their results are "clinically acceptable"? Would it be allowed to use this threshold to distribute synthetic data generated by such a model? Or is there still a chance of memorization?

We agree that these are all important questions, we have been communicating with a data protection officer about the perspectives of synthetic data since this project started. Based on preliminary discussions, our method could greatly simplify data sharing between institutes and decrease privacy risks. However, currently, there are no official, internationally accepted, guidelines as to what constitutes memorization (of imaging data).

Therefore, it is difficult to say if our approach is "clinically acceptable", given lack of prior work on what that entails. We are confident in our analysis but it could be made more robust by using several dissimilar embedding models, or by increasing the threshold. Still, our method relies on empirical performance of neural networks, and therefore no mathematical guarantees can be given that no memorization occurs. Nonetheless, we believe that empirical evaluation similar to ours could be enough for clinical acceptance, once the guidelines are determined.

As to the immediate implications, we believe our method to be beneficial, since the current standard practice seems to be to release synthetic data or generative models without any concern for memorization (e.g., [2, 3]). 

_(the response is continued in Part 3 due to character limit)_
We thank the reviewer for their valuable feedback.

> “However, the real target would be to show that training on synthetic data is not significantly worse than training on all the real data available.”

Achieving the performance of central training with all the real data without sharing the data would be ideal. However, the assumption of this research is that we _cannot_ bring real data together (as specified in section 3.1). Under this assumption, the practically valuable goal is to improve upon the baseline of using only local real data, which we achieve.

> “It is not stated, only implied: is it correct that real-all is always significantly better than all other models?”

This is generally correct, which we acknowledged in Section 4.3.1 (“While training on real data centrally (real-all) gives the best results overall, our method comes close”). The only exception is Polyp A, where syn-real-A is better by 0.3 Dice score, which is not a meaningful improvement. While we do not achieve the upper-bound results of the real-all performance, we believe that improving upon real-A and real-B is useful in practice, with further potential improvements left for future research.

> “Using the pre-processing steps of nnUNet is certainly advantageous, however selecting the hyperparameters of the encoder and the decoder is more questionable. Although the feature space sizes of the encoder/decoder of a UNet and the generator/discriminator of a GAN are similar, their objectives are entirely different. The reviewer thinks it does not "appear natural to reuse the hyperparameters" determined by the nnUNet in the UNet architecture. Did the authors perform experiments to show that this is a reasonable decision?”

We agree that the objectives of the encoder/decoder of a UNet and the discriminator/generator of a GAN are dissimilar when considered concretely, however we argue that they are, in fact, similar when considered at a higher level of abstraction. To improve clarity, we have rephrased our explanation in section 3.3, here we provide an extended version of it (and report experimental results afterwards):

The goal of a GAN-Generator is to transform a low-resolution latent representation of an image into an image. Starting from a low-resolution tensor (shape C\*H\*W, where C is large, H and W are small), a GAN-Generator needs to gradually add information in each layer. It should simultaneously increase resolution (H, W) and decrease the number of channels (C). Similarly, the goal of a U-Net-Decoder is to transform a low-dimensional latent representation of an image into a mask (ignoring skip connections; see below). Similar to a GAN-Generator, it starts from a low-resolution tensor and needs to gradually increase the resolution and decrease the number of channels. The architectures of both GAN-Generator and U-Net-Decoder need to strike the correct balance between the speeds of increasing the resolution and decreasing the number of channels. As such, a network that strikes the correct balance in one setting, seems likely to do so in the other.

Note that we only reuse the nnU-Net parameters related to this “speed of transformation” (kernel sizes, strides, network depths), and leave the rest of the StyleGAN2 architecture that is GAN-specific (e.g., the structure of synthesis layers) untouched. We acknowledge that the mapping between the GAN and the U-Net is imperfect (e.g., there are no skip connections in the GAN) but we hope we were able to better explain why the transfer of architectural parameters from one to the other appears natural to us.

(Analogously, both the U-Net-Encoder and the GAN-Discriminator need to gradually decrease the spatial resolution and increase the number of channels to arrive at a low-dimensional representation)

_(the response is continued in Part 2 due to character limit)_
We thank the reviewer for their valuable feedback.

> “While the experimental results demonstrated the benefit of the proposed work to merge synthetic data for training, it’d be good to see comparisons to other image synthesis and federated learning works to provide empirical evidence for the limitations discussed in the related works section.”

To compare our approach to federated learning, we run two federated learning baselines: Federated Averaging (FedAvg) [1] (in combination with nnU-Net) and Distributed Synthetic Learning (DSL) [2], a state-of-the-art approach to training a GAN in a federated manner to be used for generating synthetic data and training a U-Net. Since the key contribution of our method is its automatic and hyperparameter-free nature, we run the baselines in a similar setting of no hyperparameter tuning (for FedAvg, we used 1000 communication rounds with 1 epoch of local training in between since it was the best setting in [1]; for DSL, we followed the paper and the official code base (https://github.com/tommy-qichang/DSL_All_Code), using the median values for hyperparameters that differed across the three datasets in DSL: λ-L1: 300, 150, 100 → 150; batch size: 6, 6, 3 → 6). 

The results of the experiments are reported in the Appendix D. FedAvg on average has 5.3 worse Dice Score (DS) (on the scale 0-100) than our HyFree-S3 in the i.i.d. setting of Lung, with the gap widening further in the non-i.i.d settings: 10.7 DS for Cervix, 29.2 DS for Polyp. DSL performs substantially worse in all settings (on average over 5 folds: Lung: 32.8 DS worse, Cervix: 35.1 DS worse, Polyp: 36.9 DS worse). The poor performance of DSL was unexpected, given the excellent results reported in [2]. We attribute this to untuned hyperparameters. We have checked that it learns to generate relatively realistic images (see Figure 8), on which the U-Net that it trains performs well, however it generalizes to real test images poorly. We additionally did minimal manual hyperparameter tuning of DSL in the Polyp setting and were able to improve its performance by 9.2 DS on average (see Appendix D for details).

While the performance of FedAvg and DSL could potentially be improved further via hyperparameter tuning, their default performance is subpar, highlighting the benefit of automatic methods such as HyFree-S3, the hyperparameter-free nature of which is its key benefit.

We updated our code to include the FedAvg implementation: https://github.com/AwesomeLemon/hyfree-s3-midl-review. For the DSL experiments, see our fork of the official code base: https://github.com/AwesomeLemon/DSL_All_Code.  

We hope that we have addressed the weakness highlighted by the reviewer, and we would be glad to answer any other question the reviewer may have.

[1] McMahan, B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. y. (2017). Communication-Efficient Learning of Deep Networks from Decentralized Data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 1273–1282. https://proceedings.mlr.press/v54/mcmahan17a.html

[2] Chang, Q., Yan, Z., Zhou, M., Qu, H., He, X., Zhang, H., Baskaran, L., Al’Aref, S., Li, H., Zhang, S., & Metaxas, D. N. (2023). Mining Multi-Center Heterogeneous Medical Data with Distributed Synthetic Learning. Nature Communications, 14(1). https://doi.org/10.1038/s41467-023-40687-y