Multi-scale Stochastic Generation of Labelled Microscopy Images for Neuron Segmentation

Authors: meghane  decroocq
binbin  xu
katherine l thompson-peer
adrian  moore
henrik  skibbe
b  decroocq
k l xu
a  thompson-peer
h  moore
  skibbe
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 07:05:12

Abstract: We introduce a novel method leveraging conditional generative adversarial networks (cGANs) to generate diverse, high-resolution microscopy images for neuron tracing model training. This approach addresses the challenge of limited annotated data availability, a significant obstacle in automating neuron dendrite tracing. Our technique utilizes a multi-scale cascade process to generate synthetic images from single neuron tractograms, accurately replicating the complex characteristics of real microscopy images, encompassing imaging artifacts and background structures. In experiments, our method generates diverse images that mimic the characteristics of two distinct neuron microscopy datasets, which were successfully used as training data in the segmentation task of real neuron images.

Main paper: 
 introduction: 
 The automated segmentation and tracing of neuronal structures from microscopy images is a crucial step toward understanding brain structure and function. Despite its importance, it remains a challenging task that requires high levels of morphological precision, often necessitating time-consuming manual efforts. Recent advancements in deep learning have led to substantial improvements in automatic neuron segmentation and tracing (Chen et al., 2021) . However, the effectiveness of these models is constrained by the scarcity of annotated training data and the diversity of imaging methods and neuronal morphologies.One approach to circumvent the limitations associated with the scarcity of annotated data is the generation of artificial images. This involves rasterizing a tree structure into an image and adding noise to simulate real-world imperfections. Such synthetic images are often utilized to expand existing training datasets or even to directly train models, thereby rendering the time-consuming acquisition and labeling of real images obsolete (Chen et al., 2021) . This technique is not limited to neuron segmentation but also benefits the analysis of other tree-like structures, such as vessels (Tetteh et al., 2020; Prabhakar et al., 2024) and airways (Nardelli et al., 2020) . To create images that mimic realistic morphology, these are often generated from publicly available digital reconstructions (e.g., neuron tractograms, vascular trees) (Radojević and Meijering, 2019; Chen et al., 2021) , or artificially generated trees (Hamarneh and Jassi, 2010) . Synthetic images can be generated in substantial volumes, effectively addressing the scarcity of real-world data and the imprecision of manual annotations. However, they often fail to accurately replicate the surrounding anatomical structures or imaging artifacts unique to specific organs and imaging modalities, thus limiting the transferability of the model to real-world images.In recent years, generative adversarial networks (GAN), particularly conditional GANs (cGANs) (Isola et al., 2017) , have demonstrated remarkable capabilities in generating realistic synthetic images. These models generate images conditioned on provided ground truth, creating paired images that bypass the need for further annotation. cGANs have been successfully employed across several imaging modalities, including CT (Jin et al., 2018) , MR (Lau et al., 2018; Mok and Chung, 2019) , ultrasound (Tom and Sheet, 2018) , and retinal images (Costa et al., 2017) . However, their application to microscopy imaging of neurons, to the best of our knowledge, remains unexplored. The unique challenges posed by neuron imaging include the high resolution of microscopy images, which imposes substantial computational demand and necessitates a large amount of annotated data for training. Moreover, the one-to-one image mapping inherited from cGANs constrains the diversity of the generated images to that of the input ground truth images. Consequently, achieving a broad range of image outputs requires a large and varied dataset of neuron tractograms.To address these challenges, we propose a novel method utilizing cGANs. Our approach leverages a multi-scale cGAN cascade to iteratively refine image resolution, employing a patch-based strategy that facilitates the generation of high-resolution microscopy images with minimal memory demands. An important part of our approach is the employment of a mode seeking loss function (Mao et al., 2019) that enhances the diversity of the generated data, setting our method apart from previous methods (Uzunova et al., 2019) that generate unique pairs of real and generated images. We have developed a new training strategy that ensures coherence of the image content across different scales during refinement. This allows for the stochastic generation of a broader variety of images without compromising quality. We have successfully applied our method to two datasets of neuron microscopy images, producing synthetic images that faithfully replicate the characteristics of their real counterparts. The code is available at https: //github.com/BrainImageAnalysis/Multi-scale-cGAN. 
data: 
 In this study, we use two datasets of Drosophila single-neuron microscopy images with diverse anatomical structures, and real-world imaging artifacts. Dataset 1 contains 417 widefield fluorescent microscopy images of Drosophila class I da neurons. Dataset 2 (Thompson-Peer et al., 2016; Nguyen and Thompson-Peer, 2021) contains 221 confocal microscopy images of Drosophila class IV da neurons; see Appendix A for further details. The Z-stack images were converted to 2D by maximum intensity projection, and the images of both datasets have been rescaled from 1920 × 1440 and 1024 × 1024 pixels to 1024 × 1024 pixels.Both datasets were manually traced. Neurite radii and the cell bodies were automatically extracted. All images were post-processed using histogram equalization. We downloaded all the available tracings for Drosophila class I and IV da neurons from the NeuroMorpho database (Ascoli et al., 2007) , and rasterized them into a 1024 × 1024 segmentation mask, forming the evaluation datasets Neuromorpho 1 (373 segmentations) and Neuromorpho 2 (501 segmentations).Models such as Laplacian GAN (Denton et al., 2015) or HDpix2pix (Wang et al., 2018) employ a multi-scale iterative refinement to generate high-resolution images. However, these methods impose high memory demands by requiring entire images as inputs to the network. To circumvent this limitation, we utilize a cascade of conditional GANs with a fixed input and output size. As proposed in (Uzunova et al., 2019) , rather than enlarging the size of the image processed through the model, we keep the patch size constant while iteratively decreasing the receptive field, thereby increasing the pixel resolution (Figure 1 ).Conditional GANs are generative models that learn to map a condition image x to a target image y, G : x → y. In this work, x is the neuron segmentation and y is the paired microscopy image. A discriminator D is simultaneously trained to distinguish fake from real images, driving the generator to create realistic images while following the constraints of the input image x. At the first scale, a conditional GAN (G 0 , D 0 ) learns the mapping between a lowresolution (64 × 64) neuron segmentation image x 0 and a low-resolution microscopy image y 0 , G 0 : x 0 → ỹ0 . The generated image G 0 (x 0 ) is upsampled to the original resolution (1024 × 1024) and divided into patches ỹp 0 . In the next step, another conditional GAN (G 1 , D 1 ) learns to generate next resolution patch ỹp 1 conditioned on both the segmentation x p 0 and the previous patch ỹp 0 , G 1 : (x p 1 , ỹ p 0 ) → ỹp 1 . The generated patches are combined and upsampled to the original resolution to form the refined images ỹ1 . This refinement step is repeated until we reach the target resolution. At every scale, the receptive field of the patches is divided by two; the resolution of the refined images is consequently doubled.The objective of the base generative cGAN (G 0 , D 0 ) and the refinement cGANs (G i , D i ) with i ∈ [1, n] differs. We detail the architecture of each model in the next sections.The objective of the base generative model is to create a coarse image background, laying the foundational layer for subsequent refinement at higher scales. This base model is crucial, as it determines both the realism and the diversity of the generated images.cGANs usually suffer from the lack of diversity of the generated samples for a given input, primarily because the model tends to overlook the random noise vectors. This oversight leads to a predominant one-to-one mapping between the input condition and the target image. The widely used cGAN-based pix2pix model (Isola et al., 2017) relies on a L 1 loss between the generated image G 0 (x 0 ) and the target image y 0 to improve the realism of the generated images, restricting even more the variability of the generated samples. In this type of model, the only source of variability is the input image x. To increase the diversity of the generated images, Mao et al. (Mao et al., 2019) proposed a new regularization term to drive the generator to generate dissimilar images, called mode seeking loss. We propose to use this loss in our base generation model to increase the diversity of the generated images. Our base generation cGAN is composed of a generator G 0 and a discriminator D 0 . The generator is a U-Net (Ronneberger et al., 2015) , and the discriminator a PatchGAN (Isola et al., 2017) . The generator G 0 takes as input the downsampled neuron segmentation image x 0 and a random vector z drawn from a uniform distribution. The discriminator D 0 is trained to distinguish fake pairs (x 0 , G(z, x0)) from real pairs (x 0 , y 0 ). As shown in Figure 2 , we use two losses during the training. The mode seeking loss L ms (Equation 1) maximizes the dissimilarity between two images generated from different z random vectors, increasing the diversity of the generated images and avoiding mode collapse. The adversarial loss L GAN (Equation 2) pushes the generator to create realistic images. These losses are:EQUATIONwhere z a and z b are two random vectors of size n z , and M AE is the mean absolute error, andEQUATION)The total loss L is the sum of both losses L = L ms + L GAN . Figure 3 shows the effect of the mode seeking loss on the diversity of the generated images compared to the original pix2pix model. The low-resolution images produced by the base generation cGAN are progressively enhanced in quality through a cascade of refinement cGAN models at subsequent scales. The structure of our refinement cGAN is illustrated in Figure 4 . We keep the same architecture as the base generation cGAN for the generator G i and the discriminator D i .The stochasticity of our base generation cGAN drove us to develop a new training strategy for refinement compared to existing methods. Unlike the multi-scale GAN of (Uzunova et al., 2019) , the images produced by the base cGAN do not correspond to real images. Therefore, they cannot be used as matched pairs for training the discriminator. To prevent the generator from ignoring the image generated at the previous scale (a role normally undertaken by the discriminator in cGANs), we use an M AE loss between the generated images at scales i -1 and i;EQUATION)where d is the operation of downsampling to the resolution of the previous scale, and ỹp i are the patches generated at scale i. The adversarial loss can be written as follows:L GAN (G i , D i ) = min G i max D i 2E xp i ,yp i [log D i (x p i , y p i )] + E xp i ,yp i-1 [1 -log D i (x p i , ỹp i )] (4)The total loss L is a combination of the adversarial loss and the L 1 losses, balanced by a parameter λ; L = L GAN (G i , D i ) + λL M AE (ỹ p i-1 , ỹp i ).Data augmentation (90 degree clockwise / counterclockwise rotations, horizontal and vertical flip) is applied to the images only during the base generation of scale 0. The model is trained with a batch size of 32, a learning rate of 0.002 using an Adam optimizer for all scales. Following heuristic exploration, the length of the random z vectors is set to n z = 12, and the λ parameter to 10. During the application of our method to the microscopy Datasets 1 and 2, we observed that keeping a model input patch size of 64 × 64 in the high resolutions of the cascade led to the creation of artifacts in the background of the image. We believe that it is caused by the lack of context of the patches in the high resolutions. Increasing the size of the patches to input in the model in the high-resolutions solved this problem while keeping the memory demand reasonable. The best trade-off parameters, as well as the number of iterations for each scale, can be found in Table 3 . For training and inference, we divide the images into regularly organized patches with a 50% overlap. The overlapping parts of the generated patches are merged using cosine weights, to avoid border effects in the generated images. 
image generation: 
 We present visual examples of images generated with our model. Figure 5 illustrates the ability of the refinement cGAN to enhance the details of the rather coarse image generated at scale 0. As shown in Figure 6 , our method can reproduce characteristics of the real microscopy images, such as background structures, including other surrounding neurons in Dataset 2, and imaging artifacts. Moreover, as illustrated in Figure 5 , our generative model can produce a diverse set of images from a single input neuron tractogram, while maintaining the morphology of the input neuron (more examples in Appendix D). In this section, we emphasize the contribution of the different components of our model through an ablation study. To this aim, we evaluated five models of increasing complexity;(1) pix2pix model (Isola et al., 2017) , refined according to the method of (Uzunova et al., 2019) , where the images generated at the previous scale are shown to the discriminator, (2) a hybrid model where the cGAN with mode seeking loss proposed in this work is used at Table 1 : Ablation study: our method enhances the diversity of the generated images while keeping a high fidelity to real images.The multi-scale pix2pix model (1) generates realistic images, but the model is deterministic (repeat corr = 1). As shown by the label corr values, the generated images do not strictly follow the input neuron as non-annotated branches (e.g. axons) are generated in the background. The high F ID n score of the model (2) supports our assumption that the refinement strategy of (Uzunova et al., 2019) relies on the one-to-one mapping to the real images and is not compatible with the generation of diverse images. The mode seeking loss enhances the diversity of the generated images and improves the compliance to the input neuron morphology. Finally, the L M AE loss of the proposed model helps to maintain the coherency of the generation across scales, increasing the final image diversity without compromising realism.In this section, we evaluate the applicability of our generation method for neuron segmentation, which is often a prerequisite to tracing algorithms (Zhou et al., 2018) . To this aim, we prepared four datasets; real images, images generated by our method from the training neurons of Dataset 1, images generated from the unseen neurons of NeuroMorpho 1, images generated by adding noise according to the method of (Radojević and Meijering, 2019) . We train a U-Net model to segment the neurons on a given dataset and infer on another dataset (training details in Appendix B). We compute the Dice score of the predicted segmentation to measure the tranferability between generated and real images. The results can be found in Table 2 The images generated by our method show a high transferability to real images compared to the noise-based synthetic images. The gap in Dice score compared to the training on real images is caused by some false positives in the background, missing cell bodies, and segmentation of the axons absent in the ground-truth (Figure 9 of Appendix D for visualizations). We can expect that tracing models trained on images generated using the proposed method generalize better to real data than the current models trained exclusively on noise-based synthetic data (Chen et al., 2021; Prabhakar et al., 2024) . Furthermore, our model maintains a good generation quality and diversity even when trained on small datasets (Table 5 of Appendix D), limiting the need for annotated data. Besides, as demonstrated in this study, and supported by Figure 7 and Table 4 of Appendix A, it is able to generalize to input neuron morphologies that differ from the the training set. Therefore, it can find a potential application in cases where manual tracing can hardly be performed (e.g. intricated neurons with complex branching patterns). Indeed, our model only requires segmentations to generate images, which are less burdensome to produce than tracings. The paired ground-truth tracing and microscopy images could then be created by using other available tracing data. 
conclusion and discussion: 
 We have developed a novel method utilizing a multi-scale cascade of conditional GANs to generate a diverse set of realistic, high-resolution microscopy images from neuron tractograms. Given its ability to provide accurate and rich ground-truth information, including the underlying topology and orientation of the neuronal trees, our method holds promise for facilitating more complex analyses such as tracing. Looking forward, we aim to extend our methodology to other tree-like structures, such as vessels and airways, and to explore its applicability across different imaging modalities. A limitation of our current approach is the reliance on histogram equalization of images, a pre-processing step that, while beneficial for the data used in this study, may not be suitable for all applications. In future work, we intend to explore the generation of images with intensity distributions that more closely mirror those found in real datasets. Additionally, we aim to enhance the interpretability of our generative model, striving for finer control over the generated images' characteristics.  
 
 Appendix:  


Decision: This paper presents an interesting method for multi-scale image generation. The methods are clearly presented and well motivated. The authors perform extensive experiments and insightful ablation studies. 

However, the paper remains hard to follow/unclear about the presentation of the experimental set-up: implementation details, training procedure, lack of a validation set for hyper-parameter tuning, data splits, etc. Moreover, the few given implementation/experimental details are described in a very long appendix, which further worsens readability.

Overall, I think this paper will lead to interesting discussions at MIDL, but I strongly suggest the authors to clarify their experimental set-up.

Reviews: review: paper_summary: paper_summary: Decroocq et al. propose a conditioned generative approach to generate virtual microscopy images from a given binary mask. The method builds upon previous work from Uzunova et al. 2019 (cited in the manuscript) to iteratively reconstruct the generated image using a hierarchical schema. To support the generation of different synthetic images for a given mask, they introduce a regularising random component together with the input mask as suggested by Mao et al. 2019 (cited in the manuscript). The approach is explained in detail. The method is used with two different datasets but quantitative metrics are only shown for Dataset I (NeuroMorpho dataset of Drosophila class I).
main_review: main_review: Annotated training data generation is quite an interesting approach for bioimaging. In this sense, the work of the authors becomes interesting. 
The revised version of the manuscript addresses all the points highlighted in the weaknesses of this review. Importantly, the authors' methodology and metrics to test the accuracy of the proposed method (image generation and posterior testing with a segmentation task) are convenient and a good reference for future works. 

Still, the improvement of the segmentation accuracy is unclear, so an example in which this method can be of important utility is missing, which one could agree that it is out of the scope of the current paper/conference.
strength_weakness: strength_weakness: - The proposed approach requires a tiling strategy to crop different patches of the generated images at different scales and enhance their resolution. Yet, **there are no details about how this tiling is performed**. 
   - Is the receptive field of the generator taken into consideration?
   - How is the overlap between patches performed to avoid tiling artefacts? 

- In Figure 1, it seems that in the Scale 1, a patch is cropped from the generated image and then downsampled. This downsampled patch is refined and then upsampled again to repeat the process. Is this correct? Then, why do the authors upsample the refined image, if this one is then downsampled? Also, by downsampling the patch, there is less space in pixels to improve the resolution of the generated image.
   - When training the cGANs, what's the resolution of the ground truth image? Are these downsampled accordingly?

- In the text, the authors use the concept of resolution to refer to scaling (super-pixelating) factors. Please, note that **resolution is not the size of an image in pixels**, which is given in Table 1. Also, the description in section 2.2.4 should be more clear: why does the number of training iterations change for different scales? There seems to be a relationship between the patch size of the input and output images but this is not clear in the text. Does it mean that the hierarchical steps are limited to a few upsampling steps?

- While the proposed approach looks promising, **the current work lacks proper quantitative assessment**:
    - The authors do not provide any quantitative metric to assess the quality of the generated images, nor about the training process. Because this is a conditional GAN, there are ways of computing pixel-wise metrics. Also, there are ways of measuring the quality of the generated images with respect to the ground truth domain using different perceptual metrics. 
    - The results in Table 2 cannot be interpreted without standard deviations. Indeed, looking at the three different combinations, one would conclude that the generated images do not contribute to the learning process of the network as the results for the real and generated+real are almost the same.

- **The entire segmentation pipeline should be better described**: 
    - In section 2.3, it is said "At inference, we generate 100 synthetic images per training subset from the segmentations". What does this mean? Are these the images generated to train/fine-tune the network? If this is the case, it would explain why the generated images do not contribute to the learning process, as in the end, the masks used are the same as the real ones. 
    - What does it mean that training subsets were taken? Is it that the authors take only 10 images from the training dataset and train a UNet only using those 10 images? then, why are not the 291 images taken? 
    - Also, what's the hierarchical scaling chosen for this evaluation?
questions: questions: The questions to address in the rebuttal are already specified in the previous Weaknesses section.
paper_summary: paper_summary: This work presents a method based on conditional GANs for the generation of new microscopy samples from single neuron tractograms, in an effort to address the lack of annotated data for the task of neuron dendrite tracing and segmentation. A multi-scale cascade process is adopted, iteratively increasing the resolution and refining the generated samples. The authors show generation results on two different datasets, and evaluate the effect of the addition of the synthetic samples into the training of a standard semantic segmentation model.
main_review: main_review: I am improving my rating since the authors addressed my two main concerns: (1) the lack of quantitative validation metrics for the sample generation methodology, and (2) an improved segmentation section with comparison to baselines as well as ablation studies.
strength_weakness: strength_weakness: - There is a lack of relevant baselines in the evaluation of the boost in performance when adding the synthetic samples into the segmentation training. The authors compare the enhanced (real + synthetic) setting to the vanilla real setting, but having baselines with more standard data augmentation approaches (real + data aug.) would have been important here.
- The authors refer to diversity of generated samples often in the manuscript, but do not discuss on how they assess or evaluate (qualitatively and ideally quantitatively) diversity. There is currently a lack of robust metrics to assess the quality and diversity of image generation models. Regularization in the loss function(s) that generates more diversity is one way of mitigating it, but most of the diversity claims from the authors seem to be purely qualitative. For example, in Figure 3, there is a lot of visual redundancy in the generated samples.
questions: questions: - The authors do perform experiments in order to demonstrate that the synthetic images slightly improve performance for segmentation tasks (when compared to real images only). However, one big question mark is whether they would have observed the same (or better) increase in performance simply by adding an extensive data augmentation strategy to the real samples during training? They do mention using some data augmentation for the image generation, but I am assuming that no data augmentation is used in the segmentation experiments. They cannot justify using their cGAN approach if they do not demonstrate that it offers better performance compared to the standard data augmentation techniques.
- In the refinement cGAN, why is lambda set to 0.1? Does this come from tuning?
- Similarly, in the training details, some hyperparam. choices are not explained. Are these based on previous work, or determined from hyperparam. tuning?
paper_summary: paper_summary: The authors present a novel method based on conditional GANs to create synthetic neuromorphologies. They utilize two datasets from Drosophila neuromorphology with ca. 400 and 200 images each. The authors present their method tailored to neuromorphologies, provide examples of their cGAN approach and showcase the training with synthetic and real data. The authors provide evidence, that synthetic data alone creates acceptable segmentations, however, when real data is added, it still outperforms the fully synthetic set. The authors show an interesting, but not surprising approach to increase scarce data in the field of neuroanatomy.
main_review: main_review: The authors replied and answered a lot of my questions and improved the manuscript; I am slightly now in favor of the manuscript being seen at MIDL compared to my prior rating. However, my engagement is still borderline.
strength_weakness: strength_weakness: - Weak related works neuroanatomy studies. Generative AI, also cGANs, have been used in combination of neuroimaging studies that involves MRIs. In addition, there are very recent studies that focus on neurons and dendrite segmentations. I'd recommend mentioning those.
- The new method is nicely described, however, I haven't observed any studies how the single components are necessary, i.e. no ablation study has been performed or hyperparameters systematically and quantitatively tested. The authors claim that "the generator often produced structures absent from the base images", one example of anecdotally mentioned results. I would recommend showing the effects (if necessary, use the appendix)
- The authors very often describe features and methods that they use with a rationale intransparent for the reader. Is for example the seeking loss essential? 
- The reverse neuron segmentation pipeline is nice. However, in the summary statistics in Table 2 I am missing standard deviations and more examples. For example, in Figure 6, I would like to see the segmentation quality and how this behaves on real and synthetic data.
- I am not getting why all synthetic data in Figure 6 looks alike. Is there almost no variation to existing, real images?
questions: questions: * Clear explanation and validation of the key method components
* Showing variability and neuron segmentation performance
* Elaborate on training statics and outcomes, incorporate more literature w.r.t. neuroanatomy, segmentation and synthetic data.

Comments: title: Thanks to the reviewers
Answers to Reviewer 1Ypj
Answers to Reviewer NiU1
Answers to reviewer oN5a
General answer to the reviewers
comment: We would like to thank the reviewers once again, whose comments and suggestions have enabled us to improve the original article. We have uploaded a clean version of the article, without highlights.
I thank the authors for the adjustments they made to the manuscript, and particularly the addition of evaluation metrics for the image generation, and an ablation study for the segmentation section.
Thank you for adding the extra validation, experiments and changes in the manuscript.
I thank the authors' efforts in addressing each of the comments and considerably improving the presentation and validation of their work. As shown in the new figures and proven by the quality metrics, the proposed method can accurately generate virtual images resembling the real ones.
*1. Tiling strategy* 

**Q: Receptive Field Consideration?**

**A:** The receptive field of the generator was not taken into consideration when deciding the patch size. It was set so that the resolution of the image is doubled at each scale. We have found this strategy to be a good trade-off between the difficulty of the refinement model’s task (i.e. the resolution gap between the input and output) and the number of models to train. Conducting a study of the impact of the number of scales and patch sizes would be an interesting direction for further experiments. 

**Q: Overlap Handling to Avoid Artifacts?**

**A:**  We have detailed the patch strategy employed to avoid border effect in section 2.2.4. We use a 50% overlap between patches, and the overlapping parts are merged based on cosine weights.  

**Q: Purpose of Upsampling Before Downsampling?** 

**A:** To maintain a low memory footprint, we keep the array size of inputs and outputs for the generator constant at 64^2 while increasing the pixel resolution from coarse to fine. This process effectively decreases the receptive field size relative to the entire image. Initially, a 64^2 patch represents a downscaled version of the entire 1024^2 image. As the resolution increases, the receptive field narrows with each iteration, eventually matching the 64^2 patch size in full resolution. We have modified Section 2.2.1 to clarify this point. 
Figure 1 was misleading concerning the patch upsampling and downsampling in the refinement scales. The 64^2 patches generated at scale 1 are first merged to form the generated 128^2 image. It is this generated image that is upsampled to full resolution, cropped into patches with a smaller field of view, and then each patch is downsampled to 64^2. We have modified Figure 1 to clarify this point. 

**Q: Ground Truth Image Resolution and Downsampling?** 

**A:** When training the cGANs, ground-truth images are patched and scaled to match the target resolution of the generated images, as emphasized in the modified Figure 1. 

**Q: Clarification on Resolution vs. Size**

**A:** The word resolution was misused in Table 1. Besides, both Figure 1 and Table 1 were misleading the reader into thinking that the patches were refined by using a larger output size compared to the input size in the generator. In our method, the refinement is not based on the difference in size between the input and output patches but on the difference in resolution between the input and output patches. We have clarified this point by modifying Table 1, Figure 1, and Section 2.2.1. 

**Q: Training Iterations by Scale?**

**A:** The number of training iterations was decided based on the visual observation of the generated images at different epochs. We employ a different number of iterations at scale 0 as we use different loss functions for the base generation model, and at scale 1, as it bridges the base generation scale 0 and the next refinement scales. 

*2. Quantitative assessment* 

**Q: Quantitative Metrics for Image Quality and Training?** 

**A:** The lack of quantitative assessment was indeed a limitation of our work. To address it, we proposed four metrics to measure the performances of our generation model, described in an additional section 3.2. This evaluation is supplemented by an ablation study that emphasizes the contribution of each component of our model. 

**Q: Need for Standard Deviations in Table 2?** 

**A:** We added the standard deviation values for the Dice scores in Section 3.3. The segmentation experiment was however modified, as explained hereafter. 

*3. Segmentation pipeline*

**Q: Purpose of Generating 100 Synthetic Images?**

**A :** In the original segmentation experiment, we generated 100 synthetic images from real masks to train a segmentation model. The goal was to demonstrate that our stochastic model could enhance segmentation by introducing unseen variations of the original 'real' neuron images into the training set. Although we achieved a slight improvement in segmentation results, the enhancement was not substantial enough to fully demonstrate our model's potential. Consequently, we decided to replace this experiment with an ablation study and a simpler segmentation experiment to better highlight the strengths of our model. 

**Q: Reason for Selecting Training Subsets; Hierarchical Scaling?** 

**A:** In the original segmentation experiment, our objective was to show that our method can improve the segmentation results in cases where few annotated data is available. To this aim, we trained the model on small subsets of the training set (10, 20, 100). However, in the revised version, we have decided to change this segmentation experiment to focus on the transferability of the generated images to segment real images. In this new experiment, we use the whole training fold (291) during training, as suggested in your comment. The hierarchical scaling employed is as written in the training detail section.
Thank you for your comments. We have revised our article in light of your suggestions. 

**Q : Lack of Baselines with Standard Data Augmentation?** 

**A:** We explored augmentation, finding it improved segmentation results for both real images and when applied to generated data. For training on 10 images, the Dice scores were: real only 0.619 +- 0.109, real + data augmentation 0.641 +- 0.091, generated + fine-tuning 0.642 +- 0.098, and generated + fine-tuning + data augmentation 0.652 +- 0.09. 
However, since the generated images led only to minor improvements, we decided to replace this experiment with an ablation study (Section 3.2) and another, simplified segmentation experiment (Section 3.3). We believe these new studies better highlight the novelty and advantages of our generation model. The new segmentation experiment, where a segmentation model is trained on synthetic datasets and tested on real data, demonstrates through quantitative (Dice score) and qualitative results (Figure 9, Appendix D) that models trained on generated images generalize well to real images. 

**Q: Evaluation of Sample Diversity?** 

**A :** The lack of quantitative metrics was indeed a limitation of the article. We addressed this limitation using four quantitative metrics to evaluate the proposed approach, including a metric measuring the diversity of the images created. We included an ablation study (Section 3.2), showing that the diversity of the generated images is greatly improved compared to deterministic models like pix2pix, without compromising the realism of the images. 

**Q : What About Diffusion Models?** 

**A :** Diffusion models have indeed been successfully applied to image generation in the past years and we believe that they could contribute to improving our model. To the best of our knowledge diffusion models have not yet been applied to the conditioned generation of microscopy images. The work cited in the comment employs diffusion models to generate unconditioned artificial microscopy images of microtubules. This lack of conditioning makes the ground-truth information unavailable, unlike the proposed method. Nevertheless, we have identified some recent work on conditioned diffusion models from the computer vision domain that we would like to apply to microscopy image generation in a future work [2]. 

[2] *Ho, Jonathan, et al. "Cascaded diffusion models for high fidelity image generation." Journal of Machine Learning Research 23.47 (2022): 1-33.*

**Q : More Examples of Generated Samples?** 

**A :** In addition to Figure 5, Figure 6 also shows some examples of generated images (4 generates images for each dataset) in the main text. Unfortunately, due to space limitations, we could not move the figure from the appendix to the main article. However, we added a new figure to Appendix D (Figure 8), presenting the images generated by the different models included in our ablation study. 

**Q : Application of the Method on Other Use Case in Microscopy?**

**A :** As shown in the article, our approach was able to generate realistic images from two datasets showing very different neuron morphologies and imaging artifacts. Although we have not tried yet to generate other types of microscopy images, the results look promising. In the present article, we have chosen to condition the image generation on the neuron dendrites for an application to neuron tracing. However, we could imagine different applications, for instance, in vessel segmentation by conditioning the model on the vessel masks [1], or in cell counting by conditioning it on the cell positions. 
Beyond microscopy images, we believe that the proposed method can be transferred to other imaging modalities showing tree-like structures such as MRA (vessel) and lung CT scan (airways). Attempting to generate a broader range of imaging modalities is the challenge that we will tackle next. 

[1] *Poon, Charissa, et al. "MiniVess: A dataset of rodent cerebrovasculature from in vivo multiphoton fluorescence microscopy imaging." bioRxiv (2022): 2022-07.* 

**Q : How Were the Value Lambda, and Other Hyperparameters Decided?** 

**A :** There was a mistake in the article, the original lambda parameter was set to 10. We chose this value because it roughly corresponds to the ratio between the adversarial loss and the MAE loss, giving similar importance to both losses in the refinement process. In our experiments, we have found the optimal lambda value to be between 10 and 50. These values give enough freedom for the generator to create new details and correct the mistakes of previous refinements while keeping a high coherence between scales and therefore preserving the diversity observed at scale 0. The impact of the lambda parameter on the generation is illustrated in the ablation study (Section 3.2, Table 1). The other hyperparameters were tuned by experimenting with different values heuristically. We have mentioned this in the training details section.
Thank you for your comments. We have revised our article in light of your suggestions. 

**Q : Weak Related Work on Neuroanatomy Studies using cGANs**

**A :** We have added several references to previous work employing cGANs to generate images from different modalities (MRI, CT, Ultrasound) in the introduction. 

**Q : Lack of Ablation and Quantitative Evaluation.**

**A :** As suggested, we have conducted an ablation study. We show the results in an additional section (Section 3.2), as well as visual examples of images generated by each tested model (Appendix D). This study demonstrates the contribution of the different components of our model, as well as the improvements of our approach compared to other state-of-the-art models. Besides, it enabled us to identify a redundancy between the MAE loss used in the refinement model and the training strategy consisting in refining simultaneously the real images. We have found the MAE loss to be sufficient to maintain the diversity and the realism of the generated images. We have modified the model in consequence (Figure 4, Section 2.2.3). 

**Q : Is the Mode Seeking Loss Essential?** 

**A :** We quantified the contribution of each component of the method in the ablation study of Section 3.2. In particular, we demonstrated the importance of the mode seeking loss to enhance the diversity of the generated images. 

**Q : Segmentation Experiment; Standard Deviations and Visual Examples.**

**A :** We have modified the original segmentation pipeline with a new experiment focusing on the applicability of the generated data to segment real data (Section 3.3). The standard deviations are included in the new Dice score result table (Table 2). We have also added visual examples of the segmentation predicted after training on real and synthetic datasets (Appendix D). 

**Q : Diversity of the Real and Generated Images?** 

**A :** The diversity of the generated images shown in Figure 6 mirrors the variations observed in the training data. We have quantitatively evaluated the diversity of the generated images (Table 2). Examples in Figure 7 of Appendix D demonstrate the diversity in real and generated images. 

**Q : Detailed comments : Naming of the MAE loss, Availability of the Data and Code, Optimizer.**

**A :** We have renamed the L1 loss LMAE loss as suggested, proof-read and simplified the writing of the equations in the article. The code will be available after publication of the article. We have added the corresponding github link to the end of the introduction. The datasets are not publicly available, we have cited the corresponding papers to state the provenance of the data, and provided more details on the provenance of the data (Appendix A). The optimizer used for the training is Adam, this information was added in the training details section.
We thank the reviewers for their insightful comments. Their suggestions helped us to improve the original article, especially concerning the quantitative evaluation of the proposed approach. Please find hereafter a summary of the changes that were made in the revised version. The modified parts are highlighted in red in the revised article. 

**Summary of the revisions:** 

- We clarified the explanation of the key points of our patch-based strategy (Section 2.2.1, Figure 1). 
- We designed and computed metrics to quantify the realism of the generated images, the compliance with the input neuron shape, the coherence of the generation process across scales, and the diversity of the generated images (Section 3.2) 
- We conducted an ablation study clearly demonstrating the contribution of the different components of the proposed model (Section 3.2, Table 1). The ablation study enabled us to identify and remove redundant components in our model (Section 2.2.3, Figure 4). 
- We replaced the original segmentation section by a new, simpler segmentation experiment focusing on the transferability of the generated images to segment real dataset (Section 3.3, Table 2). The new experiment, together with the ablation study, emphasizes better the advantages of the proposed approach. 
- We provided more visualizations of the generated images and the segmentation predictions, as well as additional quantitative results, in appendix (Appendix D). 
- We modified the title of the article to underline the originality of the proposed method. 

We hope that your concerns were addressed in the revised version of the article and that you will consider us for MIDL 2024 conference.