Evaluating Age-Related Anatomical Consistency in Synthetic Brain MRI against Real-World Alzheimer's Disease Data.

Authors: hadya  yassin
jana  fehr
wei-cheng  lai
alina  krichevsky
alexander  rakowski
christoph  lippert
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 13:01:44

Abstract: This study examines the realism of medical images created with deep generative models, specifically their replication of aging and Alzheimer's disease (AD) related anatomical changes. Previous research focused on developing generative methods with limited attention to image fidelity. We aim to assess the resemblance of brain MRI generated by a StyleGAN3 model with causal controls to neurodegenerative changes. For a benchmark, we conducted a visual Turing test (VTT) to see if radiologists could distinguish between synthetic and real images. Then, we employed a U-Net-based model to segment hallmarks relevant to normal aging and (AD). Finally, we conducted statistical tests for our hypothesis that no significant differences existed between real and synthetic images. (VTT) results showed radiologists struggled to differentiate between image types, highlighting (VTT)'s limitations due to subjectivity and time constraints. We found slight hippocampus distribution differences ($\textit{P}$ = 5.7e-2) and significant lateral ventricle discrepancies ($\textit{P}$s $<$ 5.0e-2), indicating higher hippocampus realism and ventricle size inconsistencies. The model more effectively simulated changes in the hippocampus than in the lateral ventricles, where difficulties were encountered with certain subgroups. We conclude that the (VTT) alone is inadequate for a comprehensive quality evaluation, promoting a more objective approach. Future research could adapt our approach to evaluate other generated medical images intended for different downstream tasks. For reproducibility, we provide detailed code implementation$^1$.

Main paper: 
 introduction: 
 Deep learning algorithms, essential for automating medical image analysis, rely heavily on access to abundant, high-quality datasets for training (Sarker, 2021) . Acquiring such data is hindered by high costs associated with equipment, expert annotation, subject availability, and privacy concerns (Diaz et al., 2021; Kaissis et al., 2020) . Consequently, these challenges can lead to data biases that significantly impair the algorithms' performance in real-world applications, as they may not accurately reflect the diversity of real patient populations or the complexity of medical conditions encountered in clinical settings (Nittas et al., 2023) .Despite significant progress in synthetic medical image generation through generative adversarial networks (GANs) (Dash et al., 2023) and Diffusion models (Yang et al., 2023) , traditional evaluation metrics like the Fréchet inception distance (FID) and Structural Similarity Index Measure (SSIM) primarily assess overall image quality, overlooking anatomical accuracy critical for medical applications. Thus, accurately mirroring human physiology and conditions like AD is crucial for synthetic images to bridge dataset gaps. Therefore, we need "Biological Plausibility" metrics, introduced by (Treder et al., 2022) and designed to provide anatomical analysis specific to clinical needs. Our evaluation method focuses on crucial neurodegenerative features for normal aging, which are more prominent in AD, including ventricular enlargement and hippocampal shrinkage (Frisoni et al., 2010; Katabathula et al., 2021) .VTT is a standard for assessing synthetic images' biological plausibility in the generative field, as utilized by Khader et al. (2022) to evaluate diffusion model performance. However, Treder et al. (2022) highlights VTT's limitations: subjectivity, high costs, and difficulty detecting subtle anatomical changes. These issues suggest that VTT is insufficient in ensuring synthetic images' clinical applicability. Our study benchmarks against VTT, promoting more objective, robust, and clinically relevant evaluation methods to improve synthetic images' clinical utility.Longitudinal GAN studies on AD explore biological plausibility. Xia et al. (2021) analyzes relative changes (RC) in the region of interest (ROI) volumes between baseline and follow-up images, real and synthetic. Peng et al. (2021) examines the ROI Absolute Volume Difference (AVD) between synthetic follow-ups and real manual segmentation. Fu et al. (2023) calculate mean absolute error (MAE) for ROI volumes in longitudinal data, differing from Ravi et al. (2022) that calculate MAE for randomly matching samples by age, sex, and clinical dementia rating (CDR). The pairwise approach in longitudinal studies, which tracks changes in the same individual, offers more accuracy than random matching. Ribeiro et al. (2023) evaluates MAE using model-predicted counterfactual volumes rather than actual volumes segmented from the counterfactual images, which may misrepresent true volumes. These methods aim to compare anatomical consistencies between real and synthetic data but reduce the evaluation to a single average value (RC, AVD, MAE), favoring benchmark comparisons of generative models over anatomical consistency and clinical relevance. Wilms et al. (2022) investigates age-related brain volume changes across generative models without real data comparison, omitting key evaluations of synthetic-to-real anatomical consistency.Additionally, the mentioned methods segment ROIs using statistical and algorithmic strategies, such as FSL (FMRIB Software Library) and FreeSurfer (Jenkinson et al., 2012; Fischl, 2012) , or multi-atlas methods (Wang et al., 2014; Doshi et al., 2013) . In contrast, our method utilizes the advanced capabilities of deep learning. Litjens et al. (2017) ; Shen et al. (2017) demonstrate that deep learning surpasses traditional methods in accuracy, efficiency, and detection of complex patterns.Deep learning segmentation enables accurate anatomical consistency comparisons between synthetic ROI sizes and real counterparts, capturing essential variability in aging and AD. Such analysis is crucial for clinical applications requiring high accuracy and transparency. Our objective is to assess the null hypothesis (H0) that there is no significant distributional difference in brain ROI areas between real Alzheimer's disease neuroimaging initiative (ADNI) images and their synthetic counterparts. We utilize statistical tests to examine differences in magnitude, certainty, direction, and distribution (dist) shapes. Finally, Jung et al. (2023) ; Xia et al. (2021) conducted statistical analyses to identify dist differences and focused specifically on anatomical consistencies across CDR groups using multi-hypothesis testing. However, our research expands to include age and sex covariates and employs an alternative approach that avoids multi-hypothesis testing. 
methodology:: 
 Figure 1 : Overview of the analysis Biological plausibility overview in Figure 1 . Input: AD-focused Synthetic images generated by a conditional GAN with causal control and a subset of the ADNI dataset as the real-world reference (unseen in training and validation of both generative and segmentation models to reduce bias). VTT was selected as a benchmark for its established role in assessing synthetic image realism. Our method quantifies ROI area via a segmentation model and performs statistical tests to evaluate anatomical consistency between image types across different covariates. Understanding our analysis requires familiarity with real-world and synthetic data characteristics, synthetic image generation, and the generative model's capabilities.Real Data. We sourced real magnetic resonance imaging (MRI) brain scans from the UK Biobank (UKB) (Sudlow et al., 2015) and ADNI (Alzheimer's Disease Neuroimaging Initiative (ADNI), 2021) datasets. The UKB provided 42,427 high-resolution T1-weighted 3T MRI brain scans of individuals aged 40-69 (mean age: 55). Unlike the UKB, which serves a general cohort, the ADNI provides a focused dataset on specific diseases. The ADNI database (adni.loni.usc.edu), established in 2003, explores the use of imaging, biological markers, and clinical assessments to track the progression of mild cognitive impairment (MCI) and early AD. ADNI consists of high-resolution T1-weighted 3T MRI brain scans from 9,183 participants aged 55-90 (mean age: 74), including 3,273 cognitive normal (CN) cases (CDR=0), 4,943 MCI cases (CDR=0.5), and 967 AD cases (CDR≥1).Synthetic Data. We generated 600,000 2D mid-slice images using a conditional stylebased generative adversarial network (StyleGAN3) model (Karras et al., 2021) with causal control (Pawlowski et al., 2020) . While Kocaoglu et al. (2017) introduced causal inference to StyleGAN3, our model combines a conditional GAN structure (Mirza and Osindero, 2014; Miyato and Koyama, 2018) with a custom causal model. To achieve a high-quality synthesis, we initially trained the model on the UKB dataset for its extensive data volume, followed by fine-tuning it on the ADNI dataset. For both datasets, our causal model consistently incorporated covariates like age, sex, and left and right lateral ventricle volumes. Given the disease-specific nature of the ADNI, we added AD-associated conditional labels like CDR and hippocampus volumes (Frisoni et al., 2010; Katabathula et al., 2021) during its fine-tuning.During the inference step, the images are synthesized by the causal and StyleGAN3 models. The causal model is used to provide conditional labels as vectors, i.e., age, sex, CDR, lateral ventricle volumes, cerebral cortex volumes, and hippocampus volumes. Taking advantage of the causal model allows us to control labels by changing ages, resulting in different volumes since our assumption is that age causally affects brain volumes. These conditional labels are fitted into the StyleGAN3 model to synthesize images with specific attributes related to age, sex, and either healthy or AD diagnoses, as indicated by CDR values, for more details on the conditional causal model (Appendix A).To assess synthetic image realism, we conducted a VTT with two radiologists reviewing 100 image pairs: 50 from UKB and 50 from ADNI with their synthetic counterparts. Each pair had one real and one synthetic image, matched by age, sex, and CDR for ADNI images only. The radiologists identified the synthetic image, rated their confidence, and explained their choices, enabling a comprehensive realism evaluation. The test, done with ImFusion Labels software (ImF), was untimed to allow breaks and resumptions.ROI Quantification via Segmentation. Our generative model produces 2D mid-slice brain MRI, while established deep-learning segmentation models like SynthSeg (Billot et al., 2023) are designed for 3D volumes. To bridge this gap, we developed a 2.5D segmentation model suitable for 2D and 3D data. Using SynthSeg, we generated ground truth masks for UKB and ADNI datasets. To prevent bias in our segmentation towards ADNI (real-world reference), we trained and validated state-of-the-art (SOTA) architectures solely on UKB.During inference, the best-performing model (adapted UNet (A-UNet)) was selected for its highest median dice similarity coefficient (DSC) from external ADNI testing. This model, an adaptation of the original UNet (Ronneberger et al., 2015) , was optimized by increasing initial features from 32 to 64 and adding a layer of depth. Details on model implementation and comparisons with other architectures are in Appendix C. We quantified ROI areas by counting pixels in segmented masks, then normalized these areas against the intracranial area to account for brain size variations. Min-max scaling was applied to these normalized values to ensure consistency with observed area ranges.Statistical Analysis. In the VTT, a two-tailed binomial test evaluated radiologists' accuracy against random guessing and Cohen's kappa for inter-rater agreement. To evaluate anatomical consistency between image types, we used statistical tests to explore the hypothesis that no significant differences exist in the ROI dists. between real and synthetic images, setting the significance level at 5.0e-2. The permutation test (permute), chosen for its non-parametric nature that requires no dist assumptions, assesses the statistical significance of mean differences through repeated recalculations, minimizing false significance risks. Subsequently, Cohen's d was used to measure effect size (ES) and direction. Additionally, we employed a 95% confidence interval (CI), based on recommendations from Lee (2016), to assess ES estimation precision and indicate the statistical significance of our findings. Finally, The Kolmogorov-Smirnov test (KS) further compared dist shapes, providing added insights into discrepancies between real and synthetic image dists. 
experiments and results: 
 Experiment Setup. Our experiments evaluated the impact of aligning covariate dists. (age, CDR, and sex) on image analysis, maintaining a 5:1 synthetic-to-real ROI area ratio throughout the study. We conducted experiments on matched and mismatched covariate dists, where we would expect a higher deviation between real and synthetic ROI dists. when deviating from matching dists. The experiments are: a) matched covariate dists, b) introduced variances by adjusting the median age of the synthetic samples' dist by ±5 years, c) CDR, and d) sex mismatches in synthetic dist are detailed in Appendix D.VTT Results. Radiologist 1 (R1) achieved a 51% accuracy rate, similar to random guessing (P = 9.2e-1), while radiologist 2 (R2) scored 15%, a significant difference (P = 4.8e-13).Cohen's kappa of 0.167 indicates low agreement between the two, suggesting variability in image perception and image fidelity conclusions. R1 often indicated "moderate certainty" and cited "contrast problems," while R2 often reported "very low certainty" and highlighted "anatomical inaccuracies" as their certainty level and reasons behind most of their decisions. Interestingly, neither of them considered an "inaccurate representation of pathology." A follow-up revealed R2's bias stemming from MRI characteristics familiarity, leading to misclassifications by associating smoothing effects with real images despite their presence in both image types. Figure 3 a ) and Figure 6 in Appendix B display a misclassification by both R2 and R1 for the same pair of images, but for different reasons. R2 stated "anatomical inaccuracies", while R1 stated "noise patterns", despite the image being real. The sole instance where both radiologists strongly agreed and accurately identified a synthetic image, citing inaccurate anatomy due to unusually small ventricles, is shown in Figure 3 b). Further VTT results are in Appendix B.Biological Plausibility Results. Regression analysis in Figure 2 reveals that Lateral Ventricles (LV) areas increase and Hippocampus (HC) areas decrease with age, aligning with neurodegeneration patterns in both real and synthetic images. Furthermore, Table 1 demonstrates significant differences for LV in matched dists. (P = 2.0e-3, negative CIlu ), indicating synthetic images overestimate LV's enlargement with age compared to real images (Negative ES, see Figure 2 ). The LV KS test results show significant shape differences in dists. across both matched and unmatched conditions (Figure 8 ). In contrast, hippocampus results show no significant difference in permute tests. However, a near-significant p (5.7e-2) and positive ES (7.0e-2) suggest minor overestimation in synthetic image ROI size reduction with age at matched dists. Analysis in Figure 2 shows minimal differences between image types in LV for the 75-80 and 80-85 age groups, with nearly identical box plots. For HC, the 60-65 and 85-90 age groups have lower medians of synthetic areas, while the 80-85 group has a higher median. .5e-1 1.3e-2 LV -3 8.2e-1 9.5e-3 -6.7e-2 8.6e-2 1.4e-2 LV Match 2.0e-3 -1.3e-1 -2.0e-1 -5.0e-2 2.0e-8 LV +3 0.0e-0 -2.3e-1 -3.1e-1 -1.6e-1 2.1e-14 LV +5 5.0e-2 -3.1e-1 -3.9e-1 -2.3e-1 4.4e-16 HC -5 4.0e-3 -1.1e-1 -1.9e-1 -3.6e-2 1.1e-2 HC -3 3.2e-1 -4.0e-2 -1.2e-01 3.7e-2 4.5e-1 HC Match 5.7e-2 7.0e-2 -6.7e-3 1.5e-1 8.7e-2 HC +3 4.0e-3 1.1e-1 3.8e-2 1.9e-1 2.4e-3 HC +5 5.0e-2 2.1e-1 1.4e-1 2.9e-1 9.4e-8Mismatched Conditions with age adjustments (+3, +5) showed significant differences in LV size, with larger negative ES than the matched dist. Conversely, deviations of (-3, -5) lacked significant findings and had smaller ES, particularly at -3, confirming the generative model's overestimation of LV sizes in matched dists. (As seen in Figure 2 ). For HC, the -3 mismatch, like matched dist, showed no significant differences but had a lower ES, suggesting minor overestimation in matched conditions due to its higher ES and a closer P value to significance. Finally, all other conditions showed significance.Outliers and the whisker ranges of box plots for both ROIs in Figure 2 show variability within and between image types, with synthetic images exhibiting a broader area size range than real ones. Closer analysis shows more outliers in real images for lateral ventricles and in synthetic images for the hippocampus. Notably, 0.82% of synthetic ventricles' areas were below the minimum observed areas in real data. These sizes are clinically significant, indicating implausibly small lateral ventricles for adults (Agreed on by R1 & R2). Figure 3 b ) illustrates such a case, clearly showing the differences in the unusual ventricle sizes of synthetic vs. real images for the same covariates, highlighting the need for rigorous evaluation of synthetic image generation for clinical use. Despite the segmentation model's training on real data, the model accurately segmented the unusually small ventricles (unseen in training), demonstrating robustness. 
discussion: 
 The VTT reveals the challenge radiologists face in distinguishing synthetic from real brain MRI, attributed to the synthetic images' compelling realism and resulting in varied accuracy values. R2's frequent misclassifications, driven by cognitive biases, hint that performance could potentially improve through retraining with varied image types. Nonetheless, the smoothness effect is not only present in synthetic images but also in real ADNI and UKB images. Additionally, R2 consistently reported low confidence, often citing 'anatomical inaccuracies' in real images identified as synthetic, emphasizing the complexity of identifying subtle changes in ROIs linked to normal aging or AD progression. Although both radiol- ogists agreed and correctly identified the unusually small ventricles, manually inspecting 600,000 synthetic images to find similar conditions is infeasible. In contrast, our method allows for efficient identification of such cases. Based on these challenges, VTT alone is insufficient as a definitive measure of image realism, necessitating a more objective approach.Our study advances this objective by rigorously evaluating the generative model's ability to replicate age-related anatomical changes, revealing its strengths and pinpointing limitations. Unlike the VTT, our method provides precise insights into anatomical accuracy, especially highlighting how the model overestimates ventricular enlargement while performing better in simulating hippocampal changes. This nuanced understanding highlights the importance of employing focused methods to evaluate the anatomical consistency of synthetic medical imagery. Finally, while observing a wide range of area sizes in synthetic images indicates no mode collapse, but outliers, particularly implausibly small ventricles, illustrates the model's current limitations in capturing the full spectrum of individual variability in real-world data.The presence of biases in synthetic data, especially those that do not accurately reflect real-world conditions, poses a significant challenge in research on aging and AD diseases. When synthetic images inaccurately represent brain ventricles as larger than expected, this introduces an unrealistic bias that can mislead studies, potentially leading to incorrect conclusions about ventricular changes. This issue is compounded in areas like AD research, where patient data may be scarce, yet it is vital that the data used reflect genuine conditions to maintain the integrity of findings. The risk extends to diagnostic accuracy, where reliance on biased synthetic images could lead to misdiagnoses or the formulation of ineffective treatment plans. Similarly, training AI systems with inaccurate images could lead to errors in clinical applications, particularly those dependent on precise identification of age-related changes. To mitigate these challenges, it is crucial to refine generative models to more faithfully represent the complexities of aging. By tackling these biases head-on, we can enhance the realism and reliability of synthetic images, thereby supporting more accurate clinical research, diagnostics, and the effective use of AI in healthcare settings. These discrepancies may stem from training with the large UKB dataset and fine-tuning on the smaller ADNI dataset. A potential direction is integrating UKB's demographic diversity with ADNI's ADspecific features, potentially improving the synthetic representation of underrepresented conditions. Additionally, advanced techniques could balance the influence of each data source, ensuring both cohorts' unique characteristics are captured in the generated images. Moreover, while causal models account for ROI volumes, the generative model's 2D training might restrict performance because of reduced contextual and spatial information. This limitation highlights the necessity for additional research as we evolve our 2D proof of concept to a more clinically relevant 3D model in future work. 
conclusion and future work: 
 In conclusion, our study highlights the difficulties in differentiating synthetic from real brain MRI, emphasizing the limitations of the VTT and advocating for more objective evaluation methods. Our analysis reveals the generative model's strength in replicating age-related anatomical changes and simulating neurodegenerative features, alongside a tendency to introduce unrealistic biases, such as overestimated ventricular enlargement or implausibly small ventricles. These insights underline the necessity for robust evaluation methods for synthetic medical images, aiming to enhance image generation and facilitate their successful application in clinical settings.While our evaluation primarily focuses on neurodegeneration, its relevance extends across various medical domains. For example, adapting our segmentation model to tumor analysis in different organs necessitates retraining for tumor segmentation. Our statistical pipeline would then be used to compare tumor shapes and sizes with those of actual tumors at various stages. In brain MRI studies, examining mid-line shifts and the progression effects of tumors on adjacent ROIs in both real and synthetic images offers a unique opportunity to discern consistencies and discrepancies between image types. Developing these evaluation methods, though challenging, is essential for the effective integration of synthetic images into clinical practice and enhancing the transparency of deep learning in such a critical field.Additionally, we provide a breakdown of the reasons each radiologist cited for their choices, particularly when they correctly identified a synthetic image (see Figure 5 b )). R1 frequently attributed their decisions to 'contrast problems,' whereas R2, biased towards associating smooth images with real ones despite this characteristic being present in both image types, leaned toward 'anatomical inaccuracies.' as their rationale. Interestingly, neither radiologist selected "Inaccurate representation of pathology" as the reason for their choice. Lastly, Figures 3 and 6 demonstrate an example where both R1 and R2 misclassified a real image as synthetic for two different reasons ("anatomical inaccuracies" and "Noise Patterns," respectively).  
 
 Appendix: In our final analysis phase, we investigated extreme outliers within the lateral ventricles and hippocampus regions to detect potential segmentation inaccuracies. Notably, errors were confined to a handful of cases in both real and synthetic images, mainly resulting from the lack of typical T1-weighted contrast inherent to each image type. Furthermore, we encountered two empty synthetic images. Nonetheless, excluding these outliers from our dataset did not influence the overall significance of our results. It appears that the observed irregular contrast patterns can be traced back to anomalies in the real dataset, which were then mirrored in some synthetic images, potentially arising from errors in scanning protocols. These discrepancies introduce segmentation challenges by creating contrast variations distinct from those in the training set. For visual examples of these anomalies and the segmentation issues they caused, see Figure 7 . significance. Furthermore, in lateral ventricles, CN-or female-predominated dists.' permute P values are non-significant, in contrast to matched, and hence more closely resemble real data than matched distributions (MCI predominant, balanced sex ratio). In Figure 9 , and Figure 10 , violin plots are more similar in the hippocampus than in the lateral ventricles, reflecting the non-significant KS P -values for the hippocampus in contrast with the lateral ventricles. The only significant KS and shape differences in the hippocampus occur in the AD-predominant dist mismatch. These results align with the age mismatch findings and suggest that certain demographic characteristics influence the model's ability to generate synthetic images that accurately reflect the diversity of real anatomical structures.The observed results highlight the significance of incorporating covariates in the training and generating synthetic medical images. Our approach, utilizing a causal model that accounts for these covariates, aims to address this need. The increased resemblance of CNpredominated dists. to real data suggests that our synthetic models may be more adept at capturing the anatomical nuances of cognitively normal subjects, possibly due to initial training on extensive assumed to be healthy (CN) UKB cohort or reflecting the varied anatomical characteristics associated with neurodegenerative conditions like MCI and AD.The closer alignment of female-predominated dists. with real data further prompts a review of the model's sensitivity to sex-specific anatomical differences. 


Decision: The authors considerably made changes to the manuscript and could convince the more critical reviewers to move from reject to accept. Three reviewers are now voting for acceptance and the topic (evaluation of synthetic images) I think is very timely and valuable for MIDL. My vote is acceptance.

Reviews: review: paper_summary: paper_summary: The key idea is to propose a method to evaluate the realism of generative models. Particularly the paper focus on T1 brain generation of T1 based on multiple inputs. The claim of the paper is Visual Turing Test is not enough to evaluate the quality of image generation especially the fidelity. The authors propose an other analysis based on the ventricule segmentation and the hypocampus segmentation size. The segmentation is trained by a U-Net using synthseg segmentation as ground truth. It shows a bias of generation based on the age (with a larger median area for the ventricles).
main_review: main_review: First, the reviewer wants to thank the authors for the interesting conversation.
The reviewer understand that the goal of this paper is not to bring a new image generation or a new segmentation algorithm but to bring a new evaluation protocol for synthetic image generation suited for Alzheimer.
The reviewer understand the choice of the authors for 2D generation. 3D generation is much more challenging than 2D generation. 
The reviewer still thinks that the 2D generation is an important limitation and the lack of comparison with other SoTA model also.

But the reviewer still acknowledges the amount of work done by the authors, and the importance of this research. Also thanks to the revision, the paper is also clearer.
The reviewer has still uncertainty in the final rating but the authors and the reviewer XzGi have convinced the reviewer of changing its mark.
strength_weakness: strength_weakness: In the content:

- Only StyleGAN v3 has been evaluated It would have been much better to also evaluate diffusion models for example the author could have use [1, 2, 3] or any other model.

-"Despite these challenges, deep generative models like generative adversarial networks
(GANs) and Diffusion models are advancing synthetic medical image generation."  For this sentence in the introduction, it would have been better to cite GAN and diffusion model  (in medical images and natural images). 

- The analysis based on the segmentation is rather biased. The segmentation and the generation are on 2D slices. The bias on the ventricle could come from the bias that some slice are less sampled even if the authors specify that only the central slice is generated and the pixel counting is divided by the cranial size. How do the authors take into account the view of the 3D model. This analysis would be more relevant for 3D generation.

- "These findings, especially the smaller synthetic ventricles that fall below the minimum area value of real data, bear clinical significance and suggest an implausible size for an adult, warranting further investigation into the model’s training and generative processes."  Does the author have an explanation why  Is it because of the generative model algorithm, the lack of data ? This is why In the reviewer's opinion, it is important to evaluate different generative models (as said above) ?

-The appendix B4 experiment is rather weird. The ground truth is synthseg and the methods are comparison with synthseg... Because we have synthseg, why do we need to train a model  and do not use directly synthseg ?

In the form;
In general, there is a lack of illustration.
- lack of a illustration at the beginning to summarize the protocol.
- It is a paper about generative models, yet we do not see in the paper a lot of generated brain. It would have been great to show examples of misclassification of the expert for example In the paper (The figure).
- The figure 5 in the appendix is not very clear, subtitles are missing next to the images. Is it the result of segmentation model? Do we see a failure of the segmentation model

- An explanation of the causal model would have been great (at least in the appendix). It is not clear based on the paper how it works

[1] https://github.com/FirasGit/medicaldiffusion
[2] https://github.com/Stability-AI/generative-models
[3] Pinaya, Walter HL, et al. "Brain imaging generation with latent diffusion models." MICCAI Workshop on Deep Generative Models. Cham: Springer Nature Switzerland, 2022.
questions: questions: See weakness section:
Some question

- Does 2D generation biases the analysis because of the lack of context, and we need 3D generation ?
- Why did the author test only StyleGanv3 ?
- The evaluation protocol is maybe too specific to Alzheimer. How to extend the protocol to other cases ? 
-  Based on the experiments the authors made, What are the possible improvements to generate better synthetic MRI ?
paper_summary: paper_summary: The authors use a generative model to generate images of the ageing brain with and without AD. They assess the age-related and disease-induced changes in the real data and compare this to the visible changes in the generated images using first a visual assessment and secondly quantitative metrics based on the volumes of segmentations of ventricle areas. They find that radiologists can be fooled by the generated images, but that volumetry of the ventricles shows slight to significant differences in the metrics assessed. They conclude that visual inspection alone isn't conclusive about the quality of generative models, but that metrics are needed that are objective and ideally causal.
main_review: main_review: This contribution was quite inspiring to read. There were many sections where I would have liked to talk with the authors and enter a discussion. I could imagine that this will be the case for the MIDL audience as well, and that's my motivation to suggest it for oral presentation.
strength_weakness: strength_weakness: You state that you set aside data for external testing to prevent bias, but then say that after iterative refinement, you arrive at the best-performing model on the external test set. This sounds like you applied intermediate models on the test set, which isn't a proper thing to do in my opinion. Perhaps the description in this section 2.4 is just too short to convey the actual procedure. In any case, this should be clarified. It is of lesser concern as it only regards the segmentation model, which in turn is only used to measure the volumes, and if it has no bias towards any of the data subsets, it will serve its purpose even if the test set was tainted through repeated use for model selection.
questions: questions: All above "weaknesses" and "detailed comments"...!
paper_summary: paper_summary: The paper proposes a full pipeline to 1) generate synthetic brain images, 2) segment the synthesized images and 3) compute several metrics to assess their quality. The authors train a generative model and generate images conditioned on a set of factors (age, sex, cortex volume, etc.) matching real images from the ADNI database. Then, they used a U-Net (trained on UKBiobank data) to segment brain structures from the real and synthetic images. Finally, the extract the volume of different brain structures in the real and synthetic images and compare them.
main_review: main_review: Thank you to the authors that did a considerable amount of modifications to the paper. I updated my rating regarding that. The goal of the paper is now clearer and I believe tackles an important topic (results evaluation for gerenerative models), although not proposing very novel methods. I noticed a few typos remaining (see my last comment), I would advice the authors to read through the paper carefully one more time before it's potential presentation in MIDL.
strength_weakness: strength_weakness: Along the paper, the main point of interest of the paper seems to shift and I could not understand the novelty of the proposed approach.
From the abstract it seems that the authors want to emphasize that they propose novel metrics for the evaluation of synthesized brain images. They state that they use existing methods for image generation and segmentation (“followed by applying a U-Net model”), but later on the attention is focused on the development of a novel segmentation model.

In the end, it is not clear what is being tested and compared. Does the observed differences between the real and synthesized data come from the generative model or from the segmentation model? In the current state of the paper, it is not possible to answer that question as nor the generative model neither the segmentation model are thoroughly tested and compared to other existing methods.

If the main point of the paper is to propose a novel segmentation method then the introduction should focus on existing segmentation methods (QuickNat [1], FastSurfer [2], SynthSeg [3]).

Then the proposed method should be described in the main part of the paper, not in Appendix.
Finally, the method’s results should be compared with state of the art models [1-3] and segmentation results should be displayed in the main body of the paper.

If the main point of the paper is to propose new metrics for assessing the quality of synthesized images, then it should cite relevant literature in which some of the proposed metrics have already been applied (Riberio et al. [4]-> already looked at ventricular volume to assess image quality, Wilms et al. [5])

[1] Roy, A. G., Conjeti, S., Navab, N., Wachinger, C., & Alzheimer's Disease Neuroimaging Initiative. (2019). QuickNAT: A fully convolutional network for quick and accurate segmentation of neuroanatomy. NeuroImage, 186, 713-727.
[2] Henschel, L., Conjeti, S., Estrada, S., Diers, K., Fischl, B., & Reuter, M. (2020). Fastsurfer-a fast and accurate deep learning based neuroimaging pipeline. NeuroImage, 219, 117012.
[3] Benjamin Billot, Douglas N. Greve, Oula Puonti, Axel Thielscher, Koen Van Leemput,
Bruce Fischl, Adrian V. Dalca, and Juan Eugenio Iglesias. Synthseg: Segmentation
of brain MRI scans of any contrast and resolution without retraining. Medical Image
Analysis, 86:102789, 2023. ISSN 1361-8415. doi: 10.1016/j.media.2023.102789.
[4] F. D. S. Ribeiro, T. Xia, M. Monteiro, N. Pawlowski, B. Glocker, in Proceedings of the 40th International Conference on Machine Learning (PMLR, 2023; https://proceedings.mlr.press/v202/de-sousa-ribeiro23a.html), pp. 7390–7425.
[5] M. Wilms et al., IEEE Transactions on Medical Imaging. 41, 2331–2347 (2022)
questions: questions: The paper needs major rewriting to clearly explain what it presents: a novel image generation algorithm? A novel segmentation method? Novel metrics for assessing the quality of synthetic images? 

The literature review of recent work should be appropriate and coherent regarding the paper main goal. 

The results should be compared to SOTA methods regarding the main goal of the paper, unless the paper can be considered as a method paper if the innovation on the generation of segmentation method is clearly explained.

Comments: title: Last Day of the Discussion Period
Replying to Few final corrections
Few final corrections
A Kind Reminder
Response to Reviewer Cavr
Response to Reviewer XzGi
Further Response to Reviewer cJk8
Response to Reviewer cJk8
General Comment
comment: Dear Reviewers,

I hope this message finds you well. As you know, today marks the final day to discuss any further concerns. We hope that we have addressed all of your lingering questions and are attentive to any last-minute comments.

We are also currently reviewing our document to correct any possible linguistic mistakes and ensure the clarity and accuracy of our submission.

We greatly appreciate your valuable time and insights and hope our revisions have positively influenced your initial rating.
Thank you for your meticulous attention to detail. I have addressed all the suggestions and am actively reviewing the entire text for any additional issues. Is there anything else we need to cover? Have we thoroughly answered all of your questions, or is there anything more? After reviewing the revised version, did you perhaps reconsider your initial rating? 

Thank you once again for your valuable time and efforts.
By reading the new version of the paper, I have noticed a few typos that remains (see some of them below):
Introduction paragraph 3: Should start with capital letter (visual Turing test)
Figure 1 arrows going to the bottom boxes are cut 
Section 2.1 paragraph 1 – problem with the first line
Section 2.1 paragraph 2 line 4: Ours should be ours
Dear all reviewers,

We are deeply appreciative of the thoughtful feedback you have provided on our manuscript. Your expertise and detailed comments have been instrumental in enhancing our work.

As we approach the **discussion deadline**, we kindly encourage any further questions or comments you might have, particularly from reviewers cJk8 and Cavr. Our aim is to ensure that we have thoroughly addressed your concerns, and therefore, we kindly ask if you could reconsider your initial ratings.

Please rest assured that we have diligently worked to incorporate your feedback and are prepared to offer any additional clarification or information that may be required.

Once again, thank you all for your significant contributions to our manuscript's enhancement.
Thank you for your constructive feedback. Based on your comments, as well as insights from others, I have extensively revised the paper. This process has been an invaluable learning opportunity and I hope you reconsider your initial rating based on the enhanced manuscript. 

## Questions To Address In The Rebuttal:

>The paper needs major rewriting to clearly explain what it presents: a novel image generation algorithm? A novel segmentation method? Novel metrics for assessing the quality of synthetic images?

Initially, we faced challenges in articulating our thoughts clearly in the paper, given its multifaceted nature. (The main goal is to develop novel metrics for assessing the quality of synthetic images. While the generative and segmentation methods were utilized as tools, they significantly impacted the work and understanding their roles are paramount ). However, the constructive guidance received has enabled me to significantly revise the manuscript. I undertook major rewrites to closely integrate all comments provided by the reviewers. 

>If the main point of the paper is to propose a novel segmentation method then the introduction should focus on existing segmentation methods (QuickNat [1], FastSurfer [2], SynthSeg [3]).

Developing accurate segmentation models was essential, though not the main focus of our study. Given that our generative model is currently a proof-of-concept limited to 2D, with future plans to expand to 3D, we found that well-established methods like SynthSeg, designed for 3D, were not directly applicable. As a workaround, we used SynthSeg to establish ground truth data. Attempts to implement QuickNat faced challenges; it underperformed due to its reliance on auxiliary variables from SynthSeg and the need for fine-tuning with expert-annotated data—resources we did not have access to (Not included in Appendix, no beneficial value to our work). FCN-Res, which is optimized for 2D images, was identified as the second-best option and is further discussed in the appendix.

>The literature review of recent work should be appropriate and coherent regarding the paper main goal.

Yes, we agree and we have expanded on specific literature on evaluation methods you mentioned (Riberio et al. and Wilms et al.) and more in the intro. 

>The results should be compared to SOTA methods regarding the main goal of the paper, unless the paper can be considered as a method paper if the innovation on the generation of segmentation method is clearly explained.

We initially aimed to compare our work with the state-of-the-art (SOTA), particularly in terms of the Visual Turing Test (VTT), but acknowledge that our writing may not have clearly conveyed this intention. Understanding this, we have thoroughly revised the paper to articulate the main goal more clearly and explain why VTT represents our chosen SOTA. This clarification, supported by a literature review in the introduction, explains why VTT is most suitable for our research objectives.
Thank you for your valuable review. As this is my first paper, I was particularly eager for it to be understood in the manner you have described. Your feedback, along with that from other reviewers, has been instrumental in my learning process, significantly improving the manuscript's quality. We all appreciate the opportunity to refine my work based on such constructive guidance and hope that all your comments have been addressed satisfactorily.

## Weaknesses:
> You state that you set aside data for external testing to prevent bias, but then say that after iterative refinement, you arrive at the best-performing model on the external test set. This sounds like you applied intermediate models on the test set, which isn't a proper thing to do in my opinion. Perhaps the description in this section 2.4 is just too short to convey the actual procedure. In any case, this should be clarified. It is of lesser concern as it only regards the segmentation model, which in turn is only used to measure the volumes, and if it has no bias towards any of the data subsets, it will serve its purpose even if the test set was tainted through repeated use for model selection.

Yes, it was too short and indeed confusing. Therefore, we amended it in Sec. 2 under "Quantification of ROIs". ( Trained and validated solely on UKB for peak performance in internal testing. A-UNet excelled in UKB internal and ADNI external tests. Had A-UNet excelled on UKB internally and FCNResNet externally on ADNI, I would have chosen FCNRes to enhance performance on ADNI, my key dataset for analysis.) 


## Detailed Comments:
> I was torn about the length of the appendices. I had no second paper in my review that deferred so much material into the appendix. On the one hand, this is an indication of the thoroughness of the research, on the other hand, I find it hard to deal with in the short amount of time reviewers have for the reviews assigned to them. I have decided not to read any of the appendices of this or any other submission and base my evaluation solely on the main sections.

I understand and appreciate your time and effort. Yes, thoroughness is essential for the research, especially given the multifaceted nature of this work.

> In 2.2.2, you speak of causal controls of labels by changing age and continue to say "since volumes are correlated with age". Strange to read about correlation here -- isn't age causing the volume change (even if through mediators and influenced by possible confounders, and therefore only indirectly)?

Couldn't agree more, this was an oversight from our side; it should have been 'causal' and we corrected it.

>In 2.5.1 you refer to a "unified dataframe" -- what's that? 

python dataframe of all the 5:1 samples in one tabular data for analysis purposes. Thank you for highlighting this. We've clarified this approach in the revised Sampling subsection, now located in Section 3 under 'Experiment Setup'


>I was puzzled to read about the sub-chance performance of the second radiologist in the VTT. It seems that he or she was perfectly able to tell the two image categories apart, just "mixed up" the categories for whichever reason. I would have liked to hear what his or her criteria were -- perhaps the generated images looked more pleasing or something? In any case, there might have been a confounder that this observer was able to latch onto, and I would have had the desire to pinpoint this.

Yes, we were confounded by space and might not have fully indulged in this aspect before, we expanded on that in the Discussion sec. 4 1st paragraph.
>The figure 5 in the appendix is not very clear, subtitles are missing next to the images. Is it the result of segmentation model? Do we see a failure of the segmentation model

We can see how it could have been done better, thank you. Thus we fixed it by adding a successful images segmentation and extra annotations and explanations to make it clearer. (Segmentation failure happened in a couple of images (3-4) and excluding them as explained in appendix didn't effect our initial statistical results.)

>An explanation of the causal model would have been great (at least in the appendix). It is not clear based on the paper how it works

Yes we agree and we added more on it in appendix a as requested.

>The evaluation protocol is maybe too specific to Alzheimer. How to extend the protocol to other cases ?

We mentioned that it is applicable but didn't demonstrate how, hence we added future work regarding this in sec. 5 paragraph 2

>Based on the experiments the authors made, What are the possible improvements to generate better synthetic MRI ?

While identifying the weaknesses and strengths of the generative model to aid in the refinement of the generative models, is primarily our aim, we have included relevant content in the last paragraph of the of Disc. sec. 4.
Thank you for your valuable feedback and your time. We appreciate your eyes for details and your constructive criticism that helped further enhance the manuscript. We tried our best to accommodate your comments. If these clarifications address your concerns, we kindly ask that you reconsider your initial rating. 

## Questions To Address In The Rebuttal:
### In the content:

> Only StyleGAN v3 has been evaluated It would have been much better to also evaluate diffusion models for example the author could have use [1, 2, 3] or any other model. Why did the author test only StyleGanv3 ? 

Thank you for raising this point; the thought has crossed my mind. However, the main goal (evaluation) was not initially clear. We have amended this in the new PDF, where we expanded on evaluation-specific literature in the introduction and demonstrated that, unlike current studies, our purpose is not to compare generative models (state-of-the-art, SOTA) but to provide a comprehensive evaluation of the anatomical consistency of synthetic brain MR images, specifically in the context of aging and Alzheimer's disease (AD), against real-world data. Additionally, there is no available public synthetic AD dataset or checkpoint to reproduce the Khader et al. (2022) paper, deeming it infeasible.

>"Despite these challenges, deep generative models like generative adversarial networks (GANs) and Diffusion models are advancing synthetic medical image generation." For this sentence in the introduction, it would have been better to cite GAN and diffusion model (in medical images and natural images).

Thank you for pointing it out, we added references in intro.

>The analysis based on the segmentation is rather biased. The segmentation and the generation are on 2D slices. The bias on the ventricle could come from the bias that some slice are less sampled even if the authors specify that only the central slice is generated and the pixel counting is divided by the cranial size. How do the authors take into account the view of the 3D model. This analysis would be more relevant for 3D generation. Does 2D generation biases the analysis because of the lack of context, and we need 3D generation ?

It could be influenced by the lack of context and we acknowledged that in discussion section. On the other hand, experts' difficulty in distinguishing between real and synthetic images, coupled with our analysis showing good performance for the hippocampus but inconsistencies in lateral ventricles for certain subgroups, suggests that the generative model holds promise but also has areas for improvement. I have incorporated a more detailed discussion on a probable source of bias, informed by our findings, in the last paragraph of the Discussion section (Section 4) in the updated PDF. The current generative adversarial network (GAN) model serves as a proof of concept, and future plans include advancing towards 3D image generation, which is more relevant and potentially more impactful for clinical applications.


>"These findings, especially the smaller synthetic ventricles that fall below the minimum area value of real data, bear clinical significance and suggest an implausible size for an adult, warranting further investigation into the model’s training and generative processes." Does the author have an explanation why Is it because of the generative model algorithm, the lack of data ? This is why In the reviewer's opinion, it is important to evaluate different generative models (as said above) ?

This is an important point and we expanded on potential bias reason in disc. sec 4 last paragraph. Importance of evaluating different generative models (answered above) and added to the expanded literature review in intro.

>-The appendix B4 experiment is rather weird. The ground truth is synthseg and the methods are comparison with synthseg... Because we have synthseg, why do we need to train a model and do not use directly synthseg ?

I rewrote parts of the scripts to make this clearer. Synthseg as well as other established statistical methods like FSL and FreeSurfer are limited to 3D and we are limited to 2D synthetic images (See sec 2.3 under ROI Quantification). It was used to generate segmentation masks to train our 2.5D approach capable of both 2 and 3D

### In the form; In general, there is a lack of illustration.

> lack of a illustration at the beginning to summarize the protocol.

True, we had one but couldn't fit it before. However,  we added a diagram under methodology sec 2 in the new pdf

>It is a paper about generative models, yet we do not see in the paper a lot of generated brain. It would have been great to show examples of misclassification of the expert for example In the paper (The figure).

 We hear you and therefore added image illustration (now fig 3) of missclassification example and unusually small ventricles as well as another example in appendix.
Thank you to all reviewers for your valuable feedback and the time you've dedicated. We have made a concentrated effort to incorporate your comments, enhancing the clarity and depth of our manuscript. If these revisions address your concerns, we kindly request you to reconsider your initial rating.

## Key Adjustments in the Rebuttal:

1. Focus on StyleGAN v3: Selected due to its significance for our study's objectives concerning Alzheimer's disease. In the revised manuscript, we've elaborated on this selection and clarified that comparing various state-of-the-art generative models isn't our priority, our emphasis is instead on providing a comprehensive evaluation of synthetic image quality and anatomical consistency.

2. Bias in 2D Slice Analysis: Addressed potential biases and acknowledged the need to move towards 3D generation for enhanced performance and clinical relevance, detailed in the Discussion section.

3. Visual Illustrations Enhanced: Added examples of misclassifications and anomalously small ventricles to better visualize our findings.

## Clarifications Made:

- Segmentation Method Choice: Justified our approach, emphasizing the constraints of applying 3D methods to our 2D work and opting for a suitable alternative.

- Causal Model and  Expanded in the Appendix 

- Application to Other Fields: Discussed the protocol's extension to other medical fields.

- Future Directions for Synthetic MRI Generation: Outlined Areas for Improvement in Generative Model Refinement.

We appreciate the opportunity for refinement based on your constructive feedback and hope the manuscript now effectively addresses the raised points.