Heterogeneous Medical Data Integration with Multi-Source StyleGAN

Authors: wei-cheng  lai
matthias  kirchler
hadya  yassin
jana  fehr
alexander  rakowski
hampus  olsson
ludger  starke
jason m millward
sonia  waiczies
christoph  lippert
Venue: MIDL_io_2024_Conference
Published: 2024-01-23 11:03:20

Abstract: Conditional deep generative models have emerged as powerful tools for generating realistic images enabling fine-grained control over latent factors. 
In the medical domain, data scarcity and the need to integrate information from diverse sources present challenges for existing generative models, often resulting in low-quality image generation and poor controllability. 
To address these two issues, we propose Multi-Source StyleGAN (MSSG). MSSG learns jointly from multiple heterogeneous data sources with different available covariates and can generate new images controlling all covariates together, thereby overcoming both data scarcity and heterogeneity.
We validate our method on semi-synthetic data of hand-written digit images with varying morphological features and in controlled multi-source simulations on retinal fundus images and brain magnetic resonance images. Finally, we apply MSSG in a real-world setting of brain MRI from different sources. Our proposed algorithm offers a promising direction for unbiased data generation from disparate sources. For the reproducibility of our experimental results, we provide [detailed code implementation](https://github.com/weslai/msstylegans).

Main paper: 
 introduction: 
 Many medical Deep Learning (DL) applications suffer from algorithmic biases, stemming from training with limited and biased data sets due to data restriction and low disease prevalence (Kazeminia et al., 2020; Bak et al., 2022) . Currently, research has employed deep generative models to create synthetic Computer Tomography scans, brain Magnetic Resonance Images (MRIs) (Frid-Adar et al., 2018; Han et al., 2018; Hong et al., 2021; Xiang et al., 2023) and retinal fundus images (Costa et al., 2017 (Costa et al., , 2018;; Zhao et al., 2017) . This offers an opportunity to fill the data availability gap in the medical domain by augmenting available data with synthetic counterparts (Kazeminia et al., 2020) . These applications generate high-resolution medical images but are not able to generate images with specific demographic and clinical characteristics. Other generative approaches use conditional Variational Autoencoders (VAEs) that learn causal relationships to infer brain MRIs with specific clinical and demographic characteristics (Reinhold et al., 2021; Jung et al., 2021) and generate counterfactual brain MRIs (Pawlowski et al., 2020) . In the medical domain, several crucial limitations arise. Generally, data sets are small, compared with natural image data sets, different studies present diverging demographic and distributional stratifications, and different sources release different sets of latent factors. Consider the example of building a generative model to synthesize brain MRIs for developing models that predict dementia. One data source may be a cohort study and collect data from a general population, with age at collection sampled uniformly between 40 and 70; due to the cohort design, Cognitive Impairment (CI) has a very low prevalence in this source. Another data source, on the other hand, may be a specific Alzheimer's Disease (AD) cohort with a very high prevalence of CI in the sampled individuals; since age is a major risk factor for AD, this second data source will likely also collect older individuals. In addition, the two studies' collection protocols will be differently designed to satisfy different aims. Hence, the available covariates for each study will only partially overlap. While both studies are highly likely to collect information on age, demographics, and sex, the first study might collect additional information on other neurological issues, while the second study might collect clinical measurements specific to AD. Indeed, this example is the very case for the UK Biobank (UKB), a UK-based population study (Sudlow et al., 2015) , and the Alzheimer's Disease Neuroimaging Initiative (ADNI) (Petersen et al., 2010) . Generally, due to privacy concerns and high acquisition costs, medical images are usually also released only in smaller studies and can not be scraped at the same scale as natural images. For example, the UKB contains one of the largest brain MRI studies to date and only acquired images of less than 50,000 individuals so far.Training reliable generative models on medical imaging data is challenging due to several factors. Limited data availability often leads to lower visual quality and incomplete coverage of the image space. Additionally, the scarcity of labels restricts control over generated images to the covariates present in a single dataset. Most existing conditional generative models cannot leverage multiple data sources simultaneously unless label overlap is perfect (Han et al., 2021) . Moreover, models trained on a single data source suffer from reduced data availability. To address these limitations, we introduce Multi-Source StyleGAN (MSSG), which is based on StyleGAN3 (Karras et al., 2021) but learns from multiple data sources concurrently. This approach increases dataset size and enables conditional image generation with all available latent covariates.We first validate MSSG on a semi-synthetic data set of hand-written digits in which we can directly control specific morphological characteristics (Castro et al., 2019) . We also investigate MSSG's behavior on real-world medical imaging data from a single source with a simulated data source split. Finally, we apply MSSG to the realistic setting of multiple sources of brain MRI data. We show that MSSG can synthesize high-quality medical images and jointly control latent structures present in those images. 
conditional stylegans: 
 Conditional Generative Adversarial Networks (GANs) generate natural images with specified categories (Mirza and Osindero, 2014) . Given the training data and labels, the labels are passed to the generator and the discriminator. In the generator, labels are concatenated with noise into a latent representation. In the discriminator, labels are fitted as input and the discriminator distinguishes synthetic images from real images based on labels. For conditional StyleGANs, the generator transforms a one-hot-vector label c into an embedding vector. This is passed to the mapping network M with a latent vector z. The mapping network produces another latent vector w and this latent vector w is passed to the synthetic network for image generation. The StyleGAN discriminator applies a conditional projection discriminator (Miyato and Koyama, 2018); see Section A for details.In this work, we focus on an extension of conditional GANs that can (i) handle mixed-type latent factors and (ii) integrate data sources that have different labels available. Here, the generator network takes in both a noise variable z ∈ R dz and a conditioning variable c ∈ R dc , which may consist of binary, multiclass (coded as a one-hot subvector), and continuous values. Analogously to the standard StyleGAN definition, the generator maps c through an embedding network and concatenates them to the noise variables z to generate the image. The discriminator predicts 1+d c output variables, with the first output denoting the standard fake/real prediction. The remaining d c outputs are predictions for each of the input variables, with appropriate loss functions for each variable (e.g., the cross-entropy loss for categorical subfeatures, and the quadratic loss for continuous subfeatures). This requires a generative model of the latent distribution, which will be learned independently from the covariates, as described in the next section.For simplicity, we will focus on the case of two sources, D 1 and D 2 . Each data source j consists of image-covariate pairs, D j = (x j i , c j i )i. The covariates consist of a shared part c j shar that is available in both sources, a unique part c j uniq that is only available in source j, and a hidden part c j hidd that is available in the other data source but not in j, i.e., c j = c j shar , c j uniq , c j hidd . The shared part denotes variables that are available in both data sets and coincide for both data sources, c 1 shar ≜ c 2 shar ∈ R q shar . However, due to distribution shift, the unconditional distribution may differ between data sources, p(c 1 shar ) ̸ = p(c 2 shar ). A typical example may be the age of each individual: different studies often collect data in different age cohorts. The remaining variables are unique to each data source and may have different dimensionality, q j uniq . These could be specific variables, such as different brain volumes, and cataracts (a disease), which are not considered in each data source but related to the shared variable. With only two data sources, c 1 uniq ≜ c 2 hidd and vice versa.Figure 2 : Latent space model of MSSG. f 1 approximates c 2 hidd from the shared covariate c j shar and vice versa. They are integrated into a joint latent space ĉj = (c j shar , ĉj hidd , c j uniq ). The generator G gets the concatenated vector from a latent noise z and the joint latent space ĉj .We assume that the conditional distribution does not shift between data sets, that is, we assume that p(c 1 hidd |c 1 shar = ξ) = p(c 2 uniq |c 2 shar = ξ) for any ξ. This assumption allows us to model the distribution between the different latent variables. Hence, we fit stochastic models f j that can sample from the conditional distributions, respectively: on source 1, we fit f 1 (c 1 shar ) = ĉ1 uniq , which can be used in source 2 to impute the missing ĉ2 hidd , and vice versa. Different parametric and non-parametric statistical methodologies can be employed for such a distribution estimation, such as structural equation modeling (Pearl, 2013; Pawlowski et al., 2020) , Bayesian methods (Saatchi and Wilson, 2017) , or GAN-based methods (van Breugel et al., 2021) . In this work, we use Maximum Likelihood Estimation (MLE) of a handdesigned parametric model for all conditional and unconditional distributions, see Section D. For all latent factors, we fit models from which we can sample, instead of only predicting the most likely outcome.Before training the MSSG, we first fit the latent space models f 1 , f 2 as described in the previous section. In this work, all conditional models are parametrized by (generalized) linear models or conditional Gaussian Mixture Models (GMMs). We adapt the standard GAN training paradigm with alternating discriminator and generator update steps. Our proposed model has a StyleGAN backbone, but in principle, it can be used with most GANs. Figure 2 shows how the concatenated vector (z, c j shar , ĉj hidd .c j uniq ) is passed to the generator G to generate images.During the generator step, we sample the available ground-truth covariates c j shar and c j uniq directly from the training data sets, for both data sources at equal proportions. We then estimate the hidden variables from the latent space models fit in the initialization stage, ĉ1 hidd = f 2 (c 1 shar ) and vice versa. We gather the latent data of both sources in a joint batch and sample z from the noise distribution. We use c and z to generate an image with the generator, Img = G(z, c shar , c uniq , ĉhidd ). We then compute predictions from the discriminator, ŷ, ĉshar , ĉuniq , ĉhidd = D(Img), where ŷ is the binary label if the image is generated or real, and propagate the loss as described in Section 2.1 to update the generator.During the discriminator step, we generate covariates from the latent-variable model described in the previous Section 2.2. The shared and unique covariates are sampled separately for each data source. We sample the hidden conditional covariates from the corresponding latent space model from the other source. We then generate synthetic images with the generator and draw real samples from both sources at equal proportions. The discriminator is again trained to both distinguish between fake and real images and to predict the correct latent covariates from the images. However, we only compute the loss over the covariates for the real images to prevent shortcut learning, where the discriminator and generator cooperate to minimize the loss without solving the training task.After training the proposed MSSG, we can generate images while controlling all covariates from different sources jointly. To draw a fully random synthetic sample from any of the available sources, we can draw a sample from the shared covariate model, c j shar and push it through latent space models f 1 , f 2 , to generate c j uniq and c j hidd . Alternatively, of course, we can directly set the variables c shar , c uniq and c hidd to any desired values. This is shown in Figure 2 , where we describe how to use the joint latent covariables with GANs. 
experimental evaluation: 
 We first validate that MSSG can properly learn from multiple sources in the toy data set MorphoMNIST that allows full control of latent factors. Subsequently, we apply our method to real-world datasets -brain magnetic resonance images and retinal fundus images-from the UK Biobank and simulate a multi-source scenario with a data split. Lastly, we explore a fully realistic multi-source setting by incorporating images from ADNI into the UKB setting. We compare our MSSG model with single-source StyleGANs, which are trained without latent space models and only on a single source.Evaluation Metrics Various image quality metrics have been proposed, including the Inception Score (Salimans et al., 2016) , Precision-Recall (Kynkäänniemi et al., 2019) , Fréchet Inception Distance (FID) (Heusel et al., 2017) , and Kernel Inception Distance (KID) (Gretton et al., 2006) . Here, we focus on the FID and report KID and further metrics, namely Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) , Structural Similarity Index Measure (SSIM), Peak Signal-To-Noise Ratio (PSNR), in Section E.Additionally, we evaluate the controllability of the latent factors in the generated images. Previous works such as the Intra-FID (Miyato and Koyama, 2018) do not apply to images with continuous labels and do not address the controllability of specific covariates. We propose a new metric, the strata prediction score, to evaluate the controllability of continuous covariates. We stratify test set samples into m marginal bins per covariate, with each marginal bin containing 33% of the total sample size to maintain adequate representation. Within each bin, we generate 15, 000 images corresponding to the test set labels. Predictions for all covariates are made using separately trained prediction models on both real and generated images. The strata prediction score is then calculated as the weighted average Pearson correlation coefficient between predicted covariates from generated and test set images across all bins. See Section C for details. Table 1 (first column) shows that at comparable sample size, our multi-source model achieves comparable image quality performance to the baseline methods while using both full data sets (which is not possible for the singlesource models) leads to a considerable improvement in image quality. In Figure 3 we explore the controllability of the different models. As expected, our multi-source models can reliably control all three covariates, with slight improvements for larger sample sizes. The singlesource baseline models can only control the respectively available covariates well but show lowto-moderate correlation with the unavailable covariates due to the correlation between thickness, intensity, and slant in the training data (see Section D.1). Qualitatively, in Figure 7 we fix two covariates and show that our model can control the remaining two variables; note that our model can jointly control the unique variables slant and intensity, even without instances in the training data with both variables concurrently available.Next, we validate MSSG in real-world scenarios with more subtle data dependencies, focusing on two medical imaging modalities: brain MRI, used in diagnosing conditions like Alzheimer's Disease; and retinal fundus images, used in diagnosing ophthalmological conditions such as cataracts, and glaucoma, or diabetic retinopathy.In the real-world setting, we use data from the UKB resource. To validate the models' controllability, we divide each dataset into two artificial sources, allowing for a comprehensive evaluation. This approach addresses the absence of covariate labels in real-world multisource settings, which we further explore in the next section.Brain MRI We use the coronal middle slice of T1-weighted brain MRI scans, aligning them with the MNI atlas. We split the data set into two sources with N = 13, 414 and N = 13, 408 samples, respectively. The first source has age and ventricle volume as covariates, which are positively correlated due to ventricle enlargement with age (Kwon et al., 2014; Nestor et al., 2008) . The second source has age and grey matter volume, which are negatively correlated (Giorgio et al., 2010; Callaert et al., 2014; Ramanoël et al., 2018) .Image quality between Source 1 and Source 2 differs strongly; our half-sized multi-source model achieves better performance than the average of the two models, and the full model outperforms all of the former models (Table 1 , second column). As shown in Figure 3 , all models only reach low-to-moderate (but comparable) controllability of the individuals' age. As in the MorphoMNIST example, ventricle and grey matter volumes can be jointly controlled moderately well with our multi-source model. Note that all three variablesage, ventricle volume, grey matter volume -are all very hard to discern from only singleslice MRI (Ballester et al., 2021) , explaining the lower controllability performance compared to MorphoMNIST.Figure 1 demonstrates the joint control of grey matter and ventricle volumes for fixed age. Independent control of ventricle volumes leads to their enlargement, while the grey matter consistently expands across every row, where ventricle and age remain constant. A subtle brightening effect at the periphery of the MR images aligns with the anatomical placement of grey matter at the brain's edge. An intriguing observation is the inverse relationship between ventricle volumes and grey matter volumes, suggesting a neurologically negative correlation.Retinal fundus images We again use age as a shared covariate. The first source includes cataract as a binary conditional covariate, the second source contains spherical power as a unique variable, representing the lens power required to correct myopia (nearsightedness) or hyperopia (farsightedness). Due to the low incidence of cataract cases in the UKB dataset (≈ 3.9%) and to prevent class imbalances, we split the data into two sources, with 60% normal and 40% cataract images. Each source has approximately 1920 images in the training set. Due to the low sample size, in both sources, we included mirrored images and ADA-style data augmentation (Karras et al., 2020a) .In Table 1 image quality for the single-source models varies and the half-sized multisource model is comparable to the better of the two, while the full multi-source model outperforms all other models. Controllability is less stable than for the previous data sets (Figure 3 ). Spherical power can be controlled moderately well for those models that include it, but both age and cataract vary more strongly. Potentially, our "half" model overspecializes on modeling the cataract phenotype but underperforms on age, while the full model strikes a more balanced performance. Figure 14 again shows that MSSG can control the variables (cataract and spherical power) jointly when the age is set to 59.Finally, we investigate a real-world multi-source setting. We incorporate ADNI (Petersen et al., 2010) , a clinical dementia dataset, with UKB (Section 3.2). From ADNI, we select covariates age, left hippocampus, and right hippocampus from SynthSeg (Billot et al., 2023) . Studies underscore the significant correlation between hippocampal volumes and cognitive function (O'Shea et al., 2016; Evans et al., 2018) , highlighting the relevance of these covariates. For the UKB cohort, we consider age, ventricle, and grey matter volumes. Notably, the ADNI cohort is older compared to UKB (≈ 74 versus ≈ 55, respectively). Table 2 compares FID scores; MSSG outperforms the other models on the joint data set, while the more specialized single-source models model their specific data sources more closely (which would be expected). Table 6 demonstrates that the multi-source model can model all respective covariates with similar performance as the specialized baseline models. In Figure 10 , we show brain MRI generated by the multi-source model jointly trained on UKB and ADNI, and Section F.2 shows more qualitative examples of generated images. 
discussion & conclusion: 
 We introduced Multi-source StyleGAN (MSSG), a conditional generative image model capable of learning from multiple disparate data sources concurrently. Our experiments demonstrate that integrating multi-source data does not compromise image quality compared to single-source generation, and it can enhance data quality by leveraging a larger dataset. Through various case studies, we validated MSSG's ability to control variables from different sources collectively, even without access to paired variables. We believe MSSG can address data scarcity and label scarcity issues in medical image data, especially for rare diseases. However, a limitation of our current method is its reliance on a hand-designed latent space model, since we wanted to ascertain that the latent model has a good fit onto the true latent distribution. Future work could explore using non-parametric general-purpose models like DECAF (van Breugel et al., 2021) as a drop-in replacement. Additionally, extending our multi-source integration approach to other generative models such as VAEs and diffusion models holds promise for future research. 
 
 Appendix: GANs are among state-of-the-art architectures in DL to generate high-dimensional data (Wang et al., 2020) . The backbone of a GAN method contains two networks, a generator network G and a discriminator network D (Goodfellow et al., 2014) . The generator produces synthetic data from a noise vector z and the discriminator distinguishes it from real data. In recent years, frameworks such as StyleGAN (Karras et al., 2019 (Karras et al., , 2020b (Karras et al., , 2021) ) and BigGAN (Brock et al., 2019) have developed GANs to generate high-quality images at high resolutions. Recent diffusion models (Sohl-Dickstein et al., 2015) have been shown to achieve promising image synthesis (Rombach et al., 2021; Song et al., 2020 Song et al., , 2021;; Ho et al., 2020) , but have slower sampling times than GANs (Dhariwal and Nichol, 2021) , require larger training data sets (Moon et al., 2022) , and can have a slow, computationally expensive training process (for example, the training time on the CelebA-64x64 dataset required around 24 hours with 16 V100 GPUs (Wang et al., 2023) ).Conditional GANs Conditional GANs (Mirza and Osindero, 2014) were introduced to generate image data conditional on categorical factors. Several studies modified and improved the performance of conditional GANs (Miyato and Koyama, 2018; Brock et al., 2019; Karras et al., 2020b Karras et al., , 2021)) . Most works only handle categorical features instead of continuous or mixed-type labels and usually ignore dependencies between input labels. One alternative is ccGANs (Ding et al., 2021 (Ding et al., , 2023)) ; however, ccGANs only tackle univariate continuous labels, such as angles and ages. Our proposed method targets multiple continuous conditional labels and the problem of missing covariates in a database.There have been several prior works on integrating multiple sources into a joint generative model but with a different focus from our work. Most prominent are works on image-to-image translation; e.g., CycleGAN (Zhu et al., 2017) is trained with cycle-consistency loss using unpaired images to translate images between two domains. DVG-Face (Fu et al., 2021) focuses on generating dual heterogeneous paired face images to preserve identity. Kang et al. (2023) proposed a method that translates medical images across domains while preserving structural information during translation. This method requires an image from the target domain and the structure and texture features from the source domain as input. Furthermore, StarGAN v2 (Choi et al., 2020) translates images between domains with a single generator and takes an image and a style code as input to increase the diversity of translated images in the target domain. MPG (Han et al., 2021) proposed a multi-attribute version of StyleGAN2 to generate pizza images with specified ingredients and views. In this approach, a pre-trained view attribute regressor is used to impute the missing values of labels. Our work differs from these approaches, as our proposed method trains on two datasets simultaneously and uses latent space models to impute missing labels.In another related research problem, image harmonization techniques (Bashyam et al., 2022; Liu et al., 2021; Beizaee et al., 2023) address domain transfer, which is caused, e.g., by different MRI scanners or data collection setups (Liu et al., 2021) . Here, techniques take images as input and transfer them to different domains, but they are not able to control the image characteristics, such as ventricle volumes. latent vector integrated with our joint conditional vector. The mapping network of the generator consists of two fully connected layers, producing another latent vector w ∈ R 512 . The discriminator, a StyleGAN2 Discriminator, is set to default configurations. Both the generator and discriminator are trained using Adam optimizers. Models were trained with ADA data augmentation to prevent overfitting. X-Flip was applied to the retinal data sets, as the data sets are small. We adapted the conditional loss function from StyleGAN3, modifying the discriminator's output to predict correct labels and disabling the mapping network in the discriminator. In preliminary experiments, we found that weighing the fake/real loss and the covariate prediction loss equally led to considerably decreased image quality. Instead, we multiply the covariate losses by a scaling parameter λ, set to 0.1 throughout our real-world experiments. For the first few iterations of training, we grow λ exponentially from 0 to its target value. This helps emphasize image quality at the start, reaching a maximum of 0.9 in semi-multisource cases and 0.1 in the true multi-source case.With the extension to the real multi-source data sets, we can express also the property of sources as an extra covariate c j source in our conditional vector c j . This allows us to explicitly sample from either of the data sources c j source instead of it being expressed by the latent noise vector z.We trained our models on 2 A100 GPUs until the convergence of the FID score on a validation set. The duration of training is data-set dependent. For MorphoMNIST, the convergences of the FID score took 8192 steps for the proposed "half" model and 22937 steps for the proposed "full" model, as the resolution of images is 32×32, and the baseline models took 19661 and 18022 steps respectively. For MRI and retinal fundus image experiments, we used a resolution of 128 × 128. The convergence of the training took 13107 steps for the proposed "half" model and 18022 for the proposed "full" model on the MRI experiments. The baseline models on single-source data sets took 29286 and 19661 steps to reach the convergence. In the retinal fundus image experiments, the proposed "half" model took 57384 steps to converge and the proposed "full" model took 47937 steps to converge. The baseline single-source models took 39601 and 30719 steps to converge.Evaluation details Quantitative evaluation consists of image quality assessment and controllability analysis. Image quality is evaluated using FID and KID, while controllability is measured through the proposed strata prediction score. In this evaluation, test sets are stratified for each covariate, divided into m = 3 marginal bins. Each marginal bin contains 33% of the total samples. With three covariates in our experiments, this results in 3 3 = 27 subsets. Regression models, specifically, ResNet50s, are employed in each stratum to predict covariates from both test sets and generated images. A total of 15, 000 images are generated for each stratum and the score is computed as the weighted (by stratum size) average of the Pearson correlation coefficients of predicted outputs from generated and test set images.For MorphoMNIST, the ResNet50 regression models yield high performance, with Pearson correlation coefficients of 0.978 for thickness, 0.996 for intensity, and 0.999 for slant.On the real-world MRI data from the UKB cohort, ResNet50 regression models achieve correlations of 0.78 for age, 0.97 for ventricle volumes, and 0.87 for grey matter volumes. In the ADNI cohort, ResNet50 models for age, left hippocampus, and right hippocampus achieve correlations of 0.964, 0.980, and 0.984 respectively. Brain MRI For model training, covariates are fitted into separate latent space models.We show here also how latent models f j approximate the covariates c j . For the UK Biobank, Figure 5 (a) shows that ventricle volumes and grey matter volumes are estimated well by the latent model with the given ages. age is rescaled to the interval [0, 1] and modeled by a beta distribution, while ventricle and grey matter volumes, conditioned on age, can be sampled together by a Gaussian mixture regression model with 10 Gaussian components.In the ADNI data set, we have three covariates, age, left hippocampus, and right hippocampus. Another 8-component GMM regression is applied to learn to sample the left hippocampus and right hippocampus. Age is again modeled by a beta distribution (with independent parameters) and the left and right hippocampus are predicted by age. Figure 5 (b) depicts the fit of the trained latent model for the left and right hippocampus, conditioned on the age.In Figure 6 (a) we illustrate how sampling from either of the two latent models leads to varying latent space distributions. Given that the UKB cohort exhibits a younger age distribution, the latent space model trained on the ADNI dataset shows a distribution shift when sampling hippocampal volumes based on the age distribution of the UKB.Retinal fundus images The shared covariate age was modeled as a rescaled beta distribution, with minimum and maximum values set based on observed data. The binary variable cataract is incorporated into a logistic regression model conditioned on age. Spherical power is modeled using a Gaussian Mixture Model regression with 13 Gaussian components, also conditioned on age. 3 (b) . The tendency of the results is similar to those of FID reported in Section 3.2. However, the source 2 model performs better than our proposed "full" model. The proposed "half" model performs comparably to the average of the source 1 and source 2 models. Table 4 (b) depicts the results of pairwise metrics. Besides the evaluation of PSNR, in which the results are similar between the proposed method and the baselines, the multi-source StyleGAN3 outperforms the single-source baselines on the evaluations of LPIPS and SSIM.True multi-source MRI Synthetic multi-source morphological digits In this section, we present additional examples generated by the multi-source GAN. These figures were produced through the interpolation of covariate values, ranging from the minimal to the maximal values, specifically within a 30% range. This approach ensures the exclusion of outliers with extreme covariate values. As depicted in Figure 7 and Figure 8 (a), the multi-source GAN adeptly controls three continuous covariates-thickness, intensity, and slant. In Figure 7 (a), the shared covariate thickness is set to the mean value of the test set, which is 2.9. Intensity and slant are modified, with images exhibiting an increase in intensity column-wise, resulting in a brighter appearance. Additionally, there is a rotation effect from left to right row-wise, ranging from -21.3 degrees to 50.5 degrees. Conversely, Figure 7 (b) demonstrates various combinations of covariates (thickness and slant) with a fixed intensity. Here, the images become thicker column-wise, and there is a rotation effect from left to right within a single row. Finally, in Figure 8 (a), the slant (covariate from Source 2) is set to a constant value, while the generated images are controlled by thickness and intensity. In this scenario, the images become thicker in a column-wise manner while intensifying from left to right within each row.F.2. Real-World Data: MRI Synthetic multi-source MRI We split UKB randomly into two data sources and trained them with our proposed method. Here, we show more generated examples from the multisource GAN. We created the following figures by interpolating the values of covariates, starting from the minimal 30% until the maximal 30%. that if ventricle volumes increase row-wise, the grey matter volumes decrease (the slits on the edge of the brain becomes wider). On the other hand, if the grey matter volumes increase column-wise, this causes the shrinkage of ventricle volumes. This is related to the anatomical nature of these two covariates in Figure 5 (a). Ventricle volumes can be controlled monotonically well. In Figure 9 (a), the age and grey matter volumes are modified. The grey matter increases horizontally from left to right. However, the age-related changes are quite small. F.3. Real-World Data: Retinal Fundus Images Synthetic multi-source Retina The following images are also generated when two covariates interpolately increase from the minimal 30% until the maximal 30%. Figure 13 describes generated retinal fundus images by controlling these three covariates independently.In Figure 13 (a) the variable cataract is set to 0 (no cataract) while the age and spherical power are modified between the interval of the minimal 30% and the maximal 30%. The brightness of images changes with the increasing spherical power. In Figure 13 No CAT.-2.5CAT.-1.0 0.5 2.0 spherical power No CAT.-2.5CAT.-1.0 0.5 2.0 spherical power (a) Generated retinal images with an age of 59 yearsNo CAT.-2.5CAT.-1.0 0.5 2.0 spherical power (b) Another example for generated retinal images with an age of 59 years  


Decision: This submission presents a novel method for generating images by combining attributes from diverse heterogeneous sources. The presented approach demonstrates value, as confirmed by the thorough experimentation on various datasets including semi-synthetic, simulated, and real-world medical images. While one reviewer expresses concerns about the evaluations against multi-source baselines and comparisons to related methods, the authors have provided clarification on the distinction between their technique and image-to-image translation.

Despite doubts over specific evaluation metrics, this work shows promise and explores a unique direction in image generation with controllable attributes. However, the authors could enhance the final version of their paper by providing more context on the selection of covariates and offering deeper insights into the experimental results. One review has been largely disregarded for its lack of depth in the scientific feedback.

For all these reasons, and with respect to the other submissions, the recommendation is towards Acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a conditional StyleGAN method that combines multiple source datasets with different data attributes to synthesize new images. The synthesized images can be controlled by unified attributes containing attributes from multiple sources. The authors evaluate their method on a simulated multi-source setting on  hand written digit, brain, and fundus imaging datasets, and a real multi-source setting on brain MRIs.
main_review: main_review: I am satisfied with most of the responses from the authors. However, the authors disregard concerns regarding the other multi-source approaches, stating them as image-to-image translation methods. The proposed method is also an image-to-image translation method based on different data attributes, where the characteristics or style of the image is translated conditioned on attributes. Furthermore, the evaluation is weak as it is compared against single source baselines with the FID/KID score (can be unreliable, as pointed out) and strata prediction score (not fully clear). Since the proposed approach is new in combining multi-source attributes, I would adhere to my original score.
strength_weakness: strength_weakness: - The motivation of combining different cohort datasets to generate is suitable to the medical field, but discussion on related works is lacking in synthesis with multiple domains [a-d] and data harmonizations [e-f] 

[a] StarGAN v2: Diverse Image Synthesis for Multiple Domains, CVPR’20

[b] Multi-attribute Pizza Generator: Cross-domain Attribute Control with Conditional StyleGAN, BMVC’21

[c] DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition, IEEE PAMI’22

[d] Structure-preserving image translation for multi-source medical image domain adaptation, Pattern Recognition’23

[e] Deep Generative Medical Image Harmonization for Improving Cross-Site Generalization in Deep Learning Predictors, JMRI’21

[f] Style Transfer Using Generative Adversarial Networks for Multi-site MRI Harmonization, MICCAI’21

[g] Harmonizing Flows: Unsupervised MR harmonization based on normalizing flows, IPMI’23



- The proposed method performed better than the single-source StyleGAN baseline. However, it is not compared with any of the multi-source approaches. Some of the above-mentioned baselines [a-f] and a few simple baselines like CycleGANs can be included for fair evaluation. 

- I understand the evaluation image generation with non-reference metrics like FID/KID. However, the benefit of medical image synthesis lies in its utilization in downstream tasks rather than solely relying on image quality metrics. Also, is it possible to generate controlled image generation such that it can be evaluated on paired metrics like SSIM or PSNR?
questions: questions: The proposed method needs more discussion on relevant literature, evaluations, and clarifications mentioned above.
paper_summary: paper_summary: Conditional deep generative models have emerged as powerful tools for generating realistic images, enabling fine-grained control over latent factors. However, in the medical domain, the scarcity of data and the need to integrate information from diverse sources pose significant challenges for existing models. These challenges often result in low-quality image generation and limited controllability. To address these issues, this paper introduces Multi-source StyleGAN (MSSG), a conditional generative image model capable of learning from multiple disparate data sources concurrently. MSSG not only overcomes data scarcity but also addresses heterogeneity by generating new images while controlling all covariates simultaneously. Furthermore, authors conduct extensive experiments on various data, validating MSSG’s ability to control variables from different sources collectively, even without access to paired variables. The proposed algorithm offers a promising direction for unbiased data generation from disparate sources.
main_review: main_review: The authors have already addressed most of my concerns. However, the explanations of the basis for selecting the covariates are still not easy for readers to understand. I hope the authors can provide an in-depth description of this issue in the final revised version.
strength_weakness: strength_weakness: 1. Pay attention to essay writing format, the formats of figures and tables in the text should be fixed as uniformly as possible. For example, why is the use of bold text inconsistent in Table 1, Table 2, and Table 3?
2. The authors only list the results of experiments, but the reason is not adequately explained.
3. In this paper, the text's messy layout and lack of visual effects such as graphs or tables make it more difficult for readers to understand and remember the information presented. For example, the proportion of text and images in Figure 3 is not harmonious.
4. The description seems not clear enough, for example, what is the basis for selecting the covariates in the article?
questions: questions: 1. Pay attention to essay writing format, the formats of figures and tables in the text should be fixed as uniformly as possible. For example, why is the use of bold text inconsistent in Table 1, Table 2, and Table 3?
2. The authors only list the results of experiments, but the reason is not adequately explained.
3. In this paper, the text's messy layout and lack of visual effects such as graphs or tables make it more difficult for readers to understand and remember the information presented. For example, the proportion of text and images in Figure 3 is not harmonious.
4. The description seems not clear enough, for example, what is the basis for selecting the covariates in the article?
paper_summary: paper_summary: Multi-source Style-GAN (MSSG) is introduced as a conditional generative image model,
showcasing its ability to simultaneously learn from multiple data sources.

The covariates  consist of three parts: a shared part available in both sources (c_shar), a
unique part exclusive to each source (c_uniq), and a hidden part available in one source
but not the other (c_hidd).

Experiments highlight MSSG  can enhance data quality and control variables
across disparate sources, offering potential solutions to data scarcity and label scarcity
issues in medical image data, particularly for rare diseases.
While MSSG relies on a hand-designed latent space model, future work may explore
alternatives like non-parametric models like DECAF and extend the multi-source
integration approach to other generative models like VAEs and diffusion models.
main_review: main_review: The authors have addressed most of the comments that were highlighted. The addition of a pseudocode and more evaluation metrics are appreciated. Therefore, I would like to stick to my original rating.
strength_weakness: strength_weakness: Section 2.2, “The remaining variables are unique to each data source” —---> To improve readability, the authors can quote examples for unique variables.

The approach requires careful handcrafting of parametric models for modeling the relationship between covariates 

Sec 2.3 is detailed but adding a training pseudo code in the Appendix will improve the readability far better.

Although the authors provide the reasoning for using MorphoMNIST, having MNIST as the first experiment dilutes the experiment section given such a strong and very interesting introduction for medical images.
questions: questions: Most existing conditional generative models cannot leverage multiple data sources simultaneously unless label overlap is perfect. —----------> Do we have a citation for this?

In Appendix D, shouldn’t it be E[blood pressure] = ?, instead of blood pressure =?

In sources where data anonymity is of concern? Are there any alternative covariates related to scanner settings, or acquisition settings to consider?

Comments: title: A kindly reminder
Further response to Reviewer WiUs
Response to Reviewer WiUs
Response to Reviewer 4eEN
Response to Reviewer mQtq
General Response
comment: Dear all reviewers:

We would like to kindly remind the reviewers that the close date of the discussion period is today. If there are any concerns or follow-up questions, we will answer them.
Dear all reviewers: 

We would like to thank all the reviewers for their thoughtful reviews and suggestions, which are invaluable for improving our paper. We would like to kindly remind the reviewers that the close date of the discussion is approaching. 
If there are any concerns or follow-up questions, we would welcome further discussion. Additionally, we sincerely hope for continued fruitful discussions with reviewers mQtq and 4eEN, ensuring that our responses address their concerns convincingly. 

We have tried our best to address the reviewers' concerns, and we remain available to clarify any lingering doubts or provide additional information.

Thank you once again for your valuable contributions to our manuscript review process.
We further add here two more responses to the reviewer, since there is not enough space left on the official comment above. We are sorry for the inconvenience.

>In Appendix D, shouldn’t it be E[blood pressure] = ?, instead of blood pressure =?

In this scenario, blood pressure is a continuous variable and sex is a binary variable.
In this model, we fit a linear regression model for blood pressure and a logistic regression model for sex. A linear regression model indeed predicts the expected value conditioned on the input variable, but in the stochastic model, the noise variable $\epsilon$ is included: $\texttt{bloodpressure} = f^1(\texttt{age}) = \alpha \texttt{age} + \beta + \epsilon$.
We have, however, $\mathbb{E}[\texttt{bloodpressure}] = \alpha \texttt{age} + \beta$.
Note the important distinction that $f^j$ is indeed a stochastic function from which we can sample, and not only make a "best-guess" approximation, as described in Section 2.2.

>In sources where data anonymity is of concern? Are there any alternative covariates related to scanner settings, or acquisition settings to consider?

In the real-world setting of MRI experiments, we have an additional covariate $c^j_\texttt{source}$, $j \in (1, 2)$, which is used to learn data sources. This is trained like a binary variable, which can distinguish between sources (see Appendix C. Implementation Details). 
Furthermore, if data sources contain different equipment settings, this can be controlled by $c^j_\texttt{source}$ to generate an image that comes from the $j$ cohort.
We thank the reviewer for their valuable comments and suggestions on our work and manuscript. We uploaded an updated version of the paper and address their comments point-by-point below.

>Section 2.2, “The remaining variables are unique to each data source” —---> To improve readability, the authors can quote examples for unique variables.

For section 2.2, we added some examples for specific variables, which exist only in some data sets and are not generally collected from every institution. For example, UK Biobank is a more general study, while ADNI is very enriched in individuals with cognitive impairment. Therefore, we used unique variables such as grey matter volumes as unique variables in UKB and hippocampal volumes as the unique variables in ADNI. In the retinal fundus case study, cataract is a disease-specific variable. These variables usually have fewer samples and do not exist in every single data source.
>The approach requires careful handcrafting of parametric models for modeling the relationship between covariates.

In our experiments, we carefully handcrafted latent space models to approximate missing covariates. This is not an end-to-end solution, but we wanted to make sure that latent variables are fitted well in different sources (More details in appendix D. Parametric Models).

However, the proposed method can be extended with a more general non-parametric model, such as DECAF [1], or also with a straightforward parametric model such as the Gaussian Mixture Model for a more approximate solution.

[1] Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar. Decaf: Generating fair synthetic data using causally-aware generative networks. Advances in Neural Information Processing Systems, volume 34, pages 22221–22233. Curran Associates, Inc., 2021.

>Sec 2.3 is detailed but adding a training pseudo code in the Appendix will improve the readability far better.

We thank the reviewer for this suggestion and added a pseudo code of the proposed method to the Section "Multi-Source StyleGAN" in Appendix B to ensure readability.

>Although the authors provide the reasoning for using MorphoMNIST, having MNIST as the first experiment dilutes the experiment section given such a strong and very interesting introduction for medical images.

We used MorphoMNIST as a simulated validation study with known generating distribution to verify that our proposed method works reliably.
While we agree that the retinal fundus and brain MRI datasets are of much higher application interest, we preferred to prioritize these stronger sanity checks over a slightly more appealing story.

>Can the multi-source technique demonstrated in this study be extended to handle datasets containing three or more sources? Although the authors primarily focused on combining two datasets as multi-source data in their performance evaluation.

Yes, our method can be generalized to $\ge 3$ data sources.
Depending on which latent variables are available in which data source, this leads to modeling choices within the latent space models.
E.g., if a variable is available in sources 1 and 2 but not in source 3, we have to decide if we want to sample it from a model derived from source 1 or from source 2, or maybe alternate between the two.
The mathematical notation would also become more verbose, so we only presented the case of two sources, as this was the focus in our experiments.

Furthermore, we included this part to a small section in the Appendix B to provide a more general aspect of the proposed method.

>Figure 4 illustrates the latent space model of the Multi-Source Style Generative Adversarial Network (MSSG) in the paper. Including this figure in the main part of the paper would greatly benefit readers [...]

We now include the figure of the latent space architecture in the main text and hope that this can help clarify the underlying concept and proposed method when reading Sections 2.2 and 2.3.
>The authors are encouraged to evaluate the model performance using additional evaluation metrics such as Inception Score and LPIPS Similarity.[...]

We now included more evaluations of our experiments with the suggested LPIPS. However, LPIPS is a pair-wise metric and compares the perceptual similarity of patches between two images. Therefore, we can only use this metric in the controlled multi-source simulation but not in the real-world multi-source experiment, because we do not have the exact labels of both sources. Table 4 in the appendix shows the results of the evaluations from LPIPS, SSIM, and PSNR. Our proposed method outperforms the single-source models in measuring the pairwise similarity between generated and real images, conditioned on the same covariates. Nevertheless, we think that pairwise metrics should be used carefully since images could look different, even though they are conditioned on the same covariates. Please also consider our answer to Reviewer mQtq.
We appreciate the reviewer for their valuable feedback. Below, we address your comments and modified our manuscript accordingly.
If apart from these you have no further concerns, we hope that you will reconsider your rating.
> Pay attention to essay writing format, the formats of figures and tables in the text should be fixed as uniformly as possible. For example, why is the use of bold text inconsistent in Table 1, Table 2, and Table 3?

We thank the reviewer for pointing this out and we modified the tables and figures in our manuscript to ensure that the formats stay uniform.

>In this paper, the text's messy layout and lack of visual effects such as graphs or tables make it more difficult for readers to understand and remember the information presented. For example, the proportion of text and images in Figure 3 is not harmonious.

The proportion of text and images in Figures is modified. Regarding the layout of our manuscript, we have a lot of experiments, including three different data sets and a real multi-source scenario, and would like to show the results and evaluations in the main manuscript.

The layout of the texts and images in the original Figure 3 is modified and is provided now in the Appendix as Figure 14 in the revised manuscript.
Furthermore, we put the figure of the model's architecture in the main text to visually clarify our proposed method. With this, we hope that readers can easily understand the concept and architecture.

>The description seems not clear enough, for example, what is the basis for selecting the covariates in the article?

For the MRI use case, we focused on cognitive impairment, a common condition of interest in neuroimaging research.
In the controlled multi-source setting, age, ventricle volumes, and grey matter volumes have all been prominently shown to be linked to cognitive impairment [1-5].
In the real-world setting, we additionally selected left/right hippocampal volumes, since they are also related to cognitive impairment.

In the retinal fundus use case, we selected age, cataract, and spherical power as the controlled covariates.
Cataract is a common condition with a high disease burden. It also leads to a blurring of retinal fundus images, making it a confounding factor, e.g., when classifying other ophthalmological diseases, and hence an interesting target to control.
We additionally selected spherical power as a proxy for the strength of myopia and hyperopia, which are of universal interest due to their high prevalence.
Age is also a common risk factor for most eye conditions.

[1] Age-related changes of lateral ventricular width and periventricular white matter in the human brain: A diffusion tensor imaging study. Neural regeneration research

[2 ] Ventricular enlargement as a possible measure of Alzheimer’s disease progression validated using the Alzheimer’s disease neuroimaging initiative database.

[3] Age-related changes in grey and white matter structure throughout adulthood. NeuroImage 

[4] Gray Matter Volume and Cognitive Performance During Normal Aging. A Voxel-Based Morphometry Study, Frontiers in Aging Neuroscience

[5] Assessing age-related gray matter decline with voxel-based morphometry depends significantly on segmentation and normalization procedures, Frontiers in Aging Neuroscience
We thank the reviewer for their valuable feedback. We provide a point-by-point response to their concerns and questions below.
>The motivation of combining different cohort datasets to generate is suitable to the medical field, but discussion on related works is lacking in synthesis with multiple domains and data harmonizations.

Thank you for pointing out the lack of discussion on related work, we added this to the background section A.
>The proposed method performed better than the single-source StyleGAN baseline. However, it is not compared with any of the multi-source approaches. Some of the above-mentioned baselines and a few simple baselines like CycleGANs can be included for fair evaluation.

We empirically evaluated the proposed method on three different types of datasets and ensured that the proposed multi-source method performed better than the single-source baseline.
The mentioned works from the reviewer are mostly related to image-to-image translation, where images are transformed from one source to another source (e.g., cats to dogs, horses to zebras, or CT scans to MRI scans).
Most of these works do not apply to our setting, where we generate images conditional on latent factors that are only partially available in each source.
>Is it possible to generate controlled image generation such that it can be evaluated on paired metrics like SSIM or PSNR?

For our experiments, we evaluated our proposed method with FID/KID and also the strata prediction score.
FID and KID evaluate the image quality and do not require paired images. We use the proposed strata prediction score to evaluate the controllability of the models, i.e., whether the generated images faithfully represent the desired input variables.

In our opinion, pairwise metrics are not an optimal evaluation setting for our specific task.
Images are created from a small set of specified factors, but even for a fixed set of factors a large perceptual variation in the output images is permissible -- the strata prediction score is a better indicator focusing only on the variation due to the latent factors.

Nevertheless, we also added pairwise metrics, namely, SSIM, PSNR, and LPIPS (Table 5 in the appendix).
In the controlled multi-source settings, we can compare generated images with real test set images, because we have the ground truth labels for each sample.

Our proposed method outperforms the single-source models across all three metrics and all three data sets.
Note that this evaluation is not possible for the real-world multi-source experiment since we do not have all ground truth covariates for each image.

> Could authors explain it clearly so that the evaluation of the controllability of the attribute in Figure 2 is better appreciated?

The proposed strata prediction score evaluates the controllability of generative models.
For multiple continuous conditions, we divide the range of each covariate into three quantiles -- for three covariates, this results in $3^3 = 27$ separate strata.
While each marginal stratum contains exactly a third of all data points, not all joint strata contain exactly $1/27$th of all data points since data needs not be uniformly distributed across the range. 
The strata prediction score then is Pearson's correlation between the predicted values from real images and the generated images within each stratum, averaged over all strata. Therefore, the range of the score is $[-1, 1]$.
The higher the score is, the better the generated images represent the correct labels.

The idea of the strata prediction score is inspired by the Intra FID (Miyato and Koyama, 2018) [1].
Intra FID addresses only categorical conditions and the reliability of FID evaluation is highly unstable for smaller sample sizes, which is a big issue when stratifying across multiple factors.

The strata prediction score applies to both categorical and continuous data, is more directly interpretable (as it shows how well each factor is controllable), and is less prone to small sample size (as correlation scores on small samples are more reliable than FID scores).

[1] cGANs with Projection Discriminator, ICLR'18.

> Please explain how it tackles rare disease cases.

Studies with a focus on rare or less common diseases usually have much smaller sample sizes than more general studies, but are also highly enriched for their respective target condition; these dataset sizes tend to be insufficient to reliably train generative image models.

Our approach can increase the total dataset size by joining multiple datasets each with its own enriched set of rare conditions.
Here, data from other sources effectively function as control cases for the rare trait.
The larger joint multi-source dataset can enable a better overall generative model, while still representing each rare condition due to its relatively high case count in the enriched source.
We note, however, that a correct application in real-world settings requires more careful analysis in future work.
We would like to thank the reviewers for their thoughtful reviews, which helped us improve the manuscript. 
We tried our best to address all comments in the answers to each reviewer individually.

In the main text, we moved the figure with the proposed model's architecture from the appendix to the main text to improve the readability and understanding of Sections 2.2 and 2.3. Furthermore, we added some examples for the unique covariates in section 2.2.
However, we needed to move some results into the appendix to fit the page limit. Furthermore, we improved the formatting of the tables and text carefully. 

Regarding the related works, we added a new section "Multi-Source/Multi-Domain Image Generation" to the Section "Background" in Appendix A. Furthermore, we added a pseudo code of the proposed method to the Section "Multi-Source StyleGAN" in Appendix B to ensure readability, and we also added a small subsection "Generalization to more than 2 data sources" to the same section.

Nevertheless, we added new evaluations to our experiments, namely using three new metrics to measure the similarity of generated and real images.
Here, we used LPIPS, SSIM, and PSNR to evaluate three different datasets. These additional results are shown in Appendix E, and our proposed method outperforms the single-source baselines on these three datasets in almost every metric.

We hope that our comments and modifications on the manuscript answer the questions from the reviewers. Detailed responses to individual reviewers are given below.