Devil is in Details: Locality-Aware 3D Abdominal CT Volume Generation for Organ Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-28 03:13:21

Abstract: In the realm of medical image analysis, self-supervised learning techniques (SSL) have emerged to alleviate labeling demands, while still facing the challenge of training data scarcity owing to escalating resource requirements and privacy constraints. Numerous efforts employ generative models to generate high-fidelity, unlabeled 3D volumes across diverse modalities and anatomical regions. However, the intricate and indistinguishable anatomical structures within the abdomen pose a unique challenge to abdominal CT volume generation compared to other anatomical regions. To address the overlooked challenge, we introduce the Locality-Aware Diffusion (Lad), a novel method tailored for exquisite 3D abdominal CT volume generation. We design a locality loss to refine crucial anatomical regions and devise a condition extractor to integrate abdominal priori into generation, thereby enabling the generation of large quantities of high-quality abdominal CT volumes essential for SSL tasks without the need for additional data such as labels or radiology reports. Volumes generated through our method demonstrate remarkable fidelity in reproducing abdominal structures, achieving a decrease in FID score from 0.0034 to 0.0002 on AbdomenCT-1K dataset, closely mirroring authentic data and surpassing current methods. Extensive experiments demonstrate the effectiveness of our method in self-supervised organ segmentation tasks, resulting in an improvement in mean Dice scores on two abdominal datasets effectively. These results underscore the potential of synthetic data to advance self-supervised learning in medical image analysis.

Main paper: 
 introduction: 
 Due to the difficulty of acquiring and hand-labeling large amounts of volumetric medical data, an increasing share of medical models are trained with self-supervised learning (SSL) techniques [16, 18, 24, 33] , which exploits complex structures in large-scale unlabeled data to enhance efficiency and effectiveness [16] . However, for the self-supervised segmentation model SSL-ALPNet [22] trained on the augmented AbdomenCT-1K dataset (which augmented with abdominal CT images generated by Medical Diffusion [14] ) over those trained on the original AbdomenCT-1K dataset [20] . Despite the synthetic data augmentation, there is a notable decrease in SSL performance across 20% augmentation rate, indicating the limited utility of sub-optimal synthetic abdominal CT data. (b) Comparison of real and synthetic abdominal CT images. The outline of the pancreas in the real image is much more discriminative than in the synthetic image generated by Medical Diffusion [14] .high-quality unlabeled data for training encounters the obstacle of scarcity. Medical volume acquisition requires significant resource investments [19, 25, 37] . Furthermore, the sensitive nature of medical images exacerbates these complexities, necessitating careful consideration of privacy preservation measures [28, 34] .To save the costs on data acquisition, unlabelled synthetic data is used to enhance SSL tasks in a cost-effective and practical way [21, 27] . Unlabelled medical volumes are usually generated by Generative Adversarial Networks (GANs) [7] or diffusion models [11] . Noteworthy studies [14, 29] present versatile generation methods succeeding in high-resolution 3D volumes of various modalities and anatomical regions, showing promising prospects for synthetic images in downstream medical research. However, challenges such as unstable training and mode collapse problems in GAN hindered the generation of large-scale, diverse, realistic volumes [13, 29, 38] . Due to the superior image generation capabilities of diffusion models, they are widely used in creating synthetic data for medical applications. These synthetic datasets often integrated into downstream tasks by either substituting real data in the training set or augmenting small training sets. However, while diffusion models [14] excel in generating realistic volumes for various modalities and anatomical regions, synthetic abdominal CT volumes fall short of expectations. Directly integrating synthetic abdominal CT volumes into downstream models will lead to performance degradation. As shown in Figure 1a , the dice scores of the self-supervised segmentation model SSL-ALPNet [22] trained on the augmented AbdomenCT-1K dataset, which is augmented with 20% of synthetic abdominal CT images generated by Medical Diffusion [14] , achieve an improvement of mean dice scores, but are mostly and unexpectedly worse on small organs such as pancreas and spleen than those trained on the original AbdomenCT-1K dataset. Therefore, it is urgent to investigate how to generate high-quality synthetic volumes of abdominal CT with realistic anatomical structures by the diffusion model.Compared to other anatomical regions, the abdomen presents a unique challenge due to its intricate and indistinguishable anatomical structures. This complexity makes the synthesis of high-fidelity 3D abdominal CT volumes particularly challenging. Synthetic abdominal CT images frequently lack the detailed anatomical structures required for clear delineation, particularly evident in blurred contours of small organs such as the pancreas. For example, Figure 1b illustrates an abdominal CT image generated by Medical Diffusion [14] , showcasing distorted and blurred anatomical details compared to real images. For example, the pancreas is challenging to distinguish in synthetic abdominal CT images. This phenomenon is attributed to generative models operating solely from a global perspective, neglecting the refinement of intricate details. Consequently, synthetic abdominal CT images with unrealistic anatomical structures may cause the model to learn biased visual representations in SSL tasks [21, 27] , resulting in inferior performance.To address the above issue, we present a locality-aware 3D abdominal CT volume generation method named Locality-Aware Diffusion(Lad), focusing on detailed organ-specific information instead of the entire image during generation. Our work comprises three phases: Latent Space Construction, Diffusion Fitting in Latent Space, and Sampling in Latent space. During the Latent Space Construction phase and Diffusion Fitting in Latent Space phase, we construct the latent space for CT image using VQ-GAN [5] to focus on abdominal structure and train a diffusion probabilistic models [11] to fit into the latent space under the guidance of the image content and structure. Specifically, to localize the regions of anatomical structures, we use Priori Extraction module to predict the mask of abdominal organ, like the pancreas, to localize the regions of anatomical structures. The predicted mask denotes the locality we focus on to refine and serves as a valuable anatomical priori in our whole generation process. To construct a latent space that fully captures the features of the original data, we designed the Locality Refinement module. This module focuses the reconstruction model on a sub-volume cropped according to the predicted mask, enhancing the fidelity of abdominal structure reconstruction and refining the quality of the latent space. When fitting Diffusion into the latent space, to extract locality information effectively, we introduce the Locality Condition Extractor 𝐸 𝑐 , which captures both content and structural perspectives from the predicted mask. The mutually complementary combination of content and structure information guides the diffusion model to learn the distribution of latent vectors better. For sampling in the latent space, we apply Locality Condition Augmentation module to expand the original maskset. Subsequently, Condition Extractor 𝐸 𝑐 extracts abundant conditions from the augmented maskset, guiding the generation of massive volumes.Our synthetic data achieves the best scores across all metrics (FID [10] , MMD [8] , MS-SSIM [30] ) for synthesis quality in quantitative comparisons and has the closest distribution to real data in qualitative comparisons on two datasets, AbdomenCT-1K [20] dataset and TotalSegmentator [31] dataset, demonstrating realism in both holistic and localized regions of abdominal CT volumes. By experimenting with treating synthetic data as real data and augmentation methods, respectively, we effectively improve the performance of the self-supervised segmentation model SSL-ALPNet [22] on the AbdomenCT-1K dataset, especially bringing performance gains in small abdominal organs like pancreas and spleen.The main contribution of our work is threefold:(  
image generation from conditions: 
 A number of papers infusing condition information into the generation process in recent years are closely related to our work. The recent ControlNet [36] incorporates fine-tuned spatial conditioning controls to Stable Diffusion [26] , a large pre-trained text-to-image latent diffusion model. Unlike ControlNet relying on first encoding an input mask to some latent space before feeding it to the diffusion model, we additionally extract topological structure features from masks to integrate more anatomical structure information. There are studies incorporating additional medical data to aid in generation tasks. Segmentation-Guided Diffusion [15] adds segmentation guidance to diffusion models by concatenating the mask channel-wise to the network input and incorporates a random mask ablation training algorithm to enable synthesis flexibility. Unlike Segmentation-Guided Diffusion needing additional mask labels, we leverage predictions of organs from a well-trained universal segmentation model [1] , used alongside unlabeled data and serving as priori knowledge to indicate regions to refine and generate highquality large amounts of samples valuable to SSL tasks. Moreover, Xu et al. [32] and Hamamci et al. [9] have introduced innovative methods for text-guided volumetric generation, leveraging 3D chest CT scans paired with radiology reports. Unlike these methods, our method does not necessitate integrating text or data from other modalities, thereby alleviating the need for additional resources. 
method 3.1 overview: 
 Lad comprises three phases: Latent Space Construction, Diffusion Fitting in Latent Space, and Sampling in Latent space. As a crucial thread running through the entire method, predicted mask of abdominal organ like the pancreas from a well-trained universal segmentation model, UniverSeg [1] 𝑓 𝑢 , localizes the regions of anatomical structures and serves as an anatomical priori where we can dig into abdominal details. We denote the predicted mask of the 𝑖 th abdominal CT volume 𝑥 𝑖 , as 𝑦 𝑖 , 𝑖 = 0, 1, ..., |𝐷 𝑡𝑟 |, 𝐷 𝑡𝑟 means the trainset of abdominal CT volumes.Central to the efficacy of this method is the utilization of a predicted abdominal organ mask generated by a well-trained universal segmentation model, UniverSeg [1] denoted as 𝑓 𝑢 . This mask serves as an anatomical prior, facilitating the localization of anatomical structures within the abdominal region. Specifically, denoting the predicted mask for the 𝑖 th abdominal CT volume 𝑥 𝑖 as 𝑦 𝑖 , where 𝑖 = 0, 1, ..., |𝐷 𝑡𝑟 | and 𝐷 𝑡𝑟 represents the training set of abdominal CT volumes. The precision of the predicted mask 𝑦 𝑖 is insignificant. In scenarios where the prediction aligns closely with ground truth, detailed features of the abdominal organ can be effectively extracted by the mask. Conversely, inaccuracies in the prediction, indicating misalignment with the ground truth, present challenges for the segmentation model in identifying the correct anatomical regions. Therefore, enhancing precision in these areas is crucial for providing discriminative information.Considering computational requirements, the diffusion model is trained and samples in the latent space of vector quantized autoencoders (VQ-GAN) [5] . So, producing high-quality reconstructions is crucial to the quality of images generated by a diffusion model modeling the latent space. We first define a loss function L glo of VQ-GAN following [14] :Given the computational demands, the diffusion model operates within the latent space of vector quantized autoencoders (VQ-GAN) [5] . The quality of the latent space directly impacts the fidelity of images generated by the diffusion model. To ensure the latent space fully captures the features of the original data, we define a loss function L glo for VQ-GAN, following [14] :L glo = L rec + L codebook + L commit + L perc + L match + L disc (1)where L rec is a reconstruction loss, L codebook is a codebook loss, L commit is a commitment loss, L perc is a perceptual loss, L match is a feature matching loss [6] and L disc is a discriminator loss.Apparently, all of these loss functions prioritize global reconstruction quality, ignoring the importance of detail reconstruction quality, particularly in anatomical structures. The significance of detail reconstruction varies across regions, where accurate depiction of anatomical structures holds paramount importance. Even if background reconstruction is excellent, blurred details of anatomical structures significantly hinder subsequent generation tasks. Thanks to the predicted mask 𝑦 𝑖 , we can localize the specific areas within volume 𝑥 𝑖 that require attention. By cropping volume 𝑥 𝑖 into a sub-volume 𝑥 𝑖 𝑠𝑢𝑏 and performing the same operation on the reconstructed volume 𝑥 𝑖 , we obtain 𝑥 𝑖 𝑠𝑢𝑏 . Introducing the Globality Loss L glo , defined by Eq (1), we now introduce the Locality Loss L 𝑙𝑜𝑐 . This loss function is tailored to direct the reconstruction model's focus towards the sub-volume 𝑥 𝑖 𝑠𝑢𝑏 extracted from volume 𝑥 𝑖 and the corresponding reconstructed sub-volume 𝑥 𝑖 𝑠𝑢𝑏 , leveraging the predicted mask 𝑦 𝑖 to enhance the detail reconstruction of abdominal structures and thereby improve the quality of the latent space. The Locality Loss is formulated as the 𝐿 1 distance between the sub-volume 𝑥 𝑖 𝑠𝑢𝑏 of the input data 𝑥 𝑖 and the corresponding sub-volume 𝑥 𝑖 𝑠𝑢𝑏 of the output data 𝑥 𝑖 :EQUATIONso, the overall objective of our reconstruction model is to minimize the loss function L:EQUATIONwhere 𝜆 is a hyperparameter that plays a crucial role in balancing between globality and locality, thus enabling fine-grained generation for comprehensive holistic reconstruction. In our experimental setup, we empirically set 𝜆 to 1.0.3.3.1 Conditional Denoising Diffusion Probabilistic Model. We first encode volume 𝑥 𝑖 into a low-dimensional latent space through VQ-GAN [5] and subsequently train a denoising diffusion probabilistic model (DDPM) [11] on the latent representation of the volume, denoted as 𝐸 𝑣𝑞 (𝑥 𝑖 ). To generate anatomical details more precisely, we incorporate the priori knowledge from mask 𝑦 𝑖 as condition information to guide the generation, instead of generating in-distribution volumes randomly. To be specific, we design Condition Extractor 𝐸 𝑐𝑜𝑛 to extract locality details as the condition signal 𝑐 to train the model to fit 𝑝 (𝐸 𝑣𝑞 (𝑥)|𝑐), which means the distribution of the latent space of abdominal CT volumes given condition 𝑐.In order to fully incorporate anatomical structure details from priori knowledge, we introduce Condition Extractor 𝐸 𝑐 to extract locality information from the predicted mask 𝑦 𝑖 into a condition vector 𝑐 𝑖 from both content and To entail condition 𝑐 𝑖 with abdominal features, we first use the encoder of the former trained VQ-GAN 𝐸 𝑣𝑞 to encode mask 𝑦 𝑖 into the latent space of abdominal CT volume 𝑥 𝑖 and then pass it through two convolution layers in order to save computation memory. We call this extraction process Content Extractor 𝐸 𝑐𝑜𝑛 and denote the output as content vector 𝑐 𝑖 𝑐𝑜𝑛 . 𝑐 𝑖 𝑐𝑜𝑛 contains image-level information such as the size, shape and location of the abdominal organ. Since 𝐸 𝑣𝑞 is trained on abdominal CT volume trainset 𝐷 𝑡𝑟 , content vector 𝑐 𝑖 𝑐𝑜𝑛 represents encoding mask 𝑦 𝑖 into an image-level latent space, without making the most of the structure information of mask label. Besides, content vector 𝑐 𝑖 𝑐𝑜𝑛 represents the global features of the entire mask. Apparently, the significance of the background cannot match up to the foreground's.Considering the above reasons, we additionally extract the organ's structure information to fully utilize the organ itself. Structure Extractor 𝐸 𝑠𝑡𝑟 is devised to extract topological features of the mask 𝑦 𝑖 by using cubical complex to represent the 𝑗 th slice 𝑦 𝑖 𝑗 of 𝑦 𝑖 ∈ R 𝐷 ×𝐻 ×𝑊 , 𝑗 = 0, 1, ..., 𝐷 -1 and then computing Betti numbers of each cubical complex. The Betti numbers, 𝛽 𝑘 counts the number of features of dimension 𝑘, where 𝛽 0 is the number of connected components, 𝛽 1 the number of loops or holes, 𝛽 2 the number of hollow voids, etc [4] . Considering the common organ's topological structure, only the first three Betti numbers is considered in our method, as Figure 3b . Thus, there are three labels in slice 𝑦 𝑖 𝑗 , the value of each label is 0 representing the background, 1 representing the organ, and 2 representing tumors. We create different cubical complexes for each label. So the final sturcture information of 𝑦 𝑖 𝑗 is represented as {𝛽 𝑘,𝑙 }, which is a one-dimension vector has 6 elements, 𝑙 meaning label value ∈ {0, 1, 2}. As for the whole volume 𝑦 𝑖 , we concatenate the topological features of all its slices and denoted as 𝑐 𝑖 𝑠𝑡𝑟 .The two extracted condition vectors, named content condition 𝑐 𝑖 𝑐𝑜𝑛 and structure condition 𝑐 𝑖 𝑠𝑡𝑟 individually, are mutually complementarity and concatenated to represent the abdominal features, which act as condition vectors to guide the generation.To generate large amounts of volumes, we apply common data augmentation techniques, including flipping, random affine, and random elastic deformation, to expand the original maskset predicted by UniverSeg [1] . We use the augmented maskset as the abundant condition source to be extracted by Condition Extractor 𝐸 𝑐 for generation guidance.Guidance. We follow classifier-free guidance [12] to guide the diffusion model so that the denoise model adjusts predictions 𝜖 𝜃 constructed viaEQUATIONwhere AbdomenCT-1K. AbdomenCT-1K Dataset [20] collects 1,112 highresolution 3D abdominal CT images from 12 medical centers, 1000 of which have manual annotations of four abdominal organs, including the liver, kidney, spleen, and pancreas. Considering the following tests of the utilization of synthetic data in downstream tasks, 800 CT images without masks are randomly selected for the training phase and sampling phase in the generation process, while the other 200 CT images are reserved for the application tests with their corresponding ground truth mask.𝑧TotalSegmentator. TotalSegmentator [31] provides 1,228 CT volumes covering 117 classes annotated by voxel, encompassing information on over 20 abdominal organs. The size of these slices ranges from 47 to 499, and some of these volumes cover limited areas of the entire abdomen. To ensure the integrity of abdomen synthesis, we charge off these incomplete volumes. Moreover, considering the downstream tests, volumes, where the liver, kidney, spleen, and pancreas don't exist, are discarded, too. In the end, 767 volumes are used in our experiments. 80% (614 volumes) of these are used as the training set and 20% (153 volumes) are used as the testing set.4.1.2 Pre-processing. In order to enhance data utility, we apply resampling techniques to both datasets, adjusting voxel spacing to [1.6, 1.6, 2.3] and [1.1, 1.1, 1.5], respectively. Following resampling, we standardize the height and width dimensions to 256. To make the most of the pancreas masks, we employ a strategic cropping method. Utilizing a sliding window mechanism with a window size of 32, we traverse the entire volume along the Z-axis, aligning with ground truth annotations. Consequently, the processed data dimensions are uniformized to 256 × 256 × 32. Furthermore, to ensure uniformity in image intensity, we truncate voxel values to the range of [-1000, 400] and subsequently normalize them to the interval [0, 1]. 
implementation details.: 
 We train the VQ-GAN for 100,000 steps. We set the compression rate as (4, 4, 4) . We let the learning rate be 3 × 10 -4 . The batch size is set as 2. Then, we train the diffusion model for 150,000 steps with 300 timesteps, the learning rate of 1 × 10 -4 , and the batch size of 20. All experiments are performed on two NVIDIA A100 GPUs.We thoroughly evaluated the quality of synthetic data using 800 synthetic volumes from AbdomenCT-1K dataset and 610 synthetic volumes from TotalSegmentator dataset, matching the size of the training set.Synthetic Data with Most Realistic Volumes. We quantitatively evaluate the realism of synthetic volumes using Fréchet Inception Distance (FID) [10] and Maximum Mean Discrepancy (MMD) [8] . Lower FID/MMD values indicate closer distributions of synthetic volumes to real ones, implying more realistic synthetic volumes. To evaluate the ability to synthesize intricate details, we additionally cropped sub-volumes from the synthesis set based on the maximum bounding box of the abdominal organ in the original maskset, calculating localized FID and localized MMD for these sub-volumes. Due to HA-GAN being trained only with volumes of size 128 3 or 256 3 , we resized the depth of volumes in the trainset from 32 to 256 and resized them back after sampling. For computing FID and MMD, we utilized a 3D ResNet model pre-trained on 3D medical images [3] to extract features, following [29] . As shown in Table 1 , HA-GAN [29] exhibits limitations in capturing the distribution of abdominal CT volumes despite its proficiency in generating highresolution 3D thorax CT and brain MRI scans. On the other hand, Medical Diffusion [14] achieves superior performance with lower FID and MMD scores across both datasets, showcasing the robust generative capabilities inherent in diffusion models. Notably, Lad outperforms the aforementioned methods across two abdominal datasets, excelling both holistically and in localized evaluations. Particularly impressive are its FID and MMD scores, which plummet to 0.0002 and 0.0003, respectively. These results underscore Table 1 : Quantitative Comparison of Synthetic Volumes on AbdomenCT-1K and TotalSegmentator Datasets. The best scores in each column are highlighted in bold. FID and MMD metrics assess the realism of synthetic volumes generated by different methods, while MS-SSIM evaluates diversity. Additionally, localized FID/MMD scores are calculated for sub-volumes cropped from the synthesis set based on the maximum bounding box of the abdominal organ in the original mask set. Our method, Lad, achieves the highest scores in these metrics from both holistic and localized perspectives, demonstrating the significant contribution of our attention to detail to the overall enhancement of volume quality.AbdomenCT-1K [20] TotalSegmentator [ Synthetic Data with Most Diverse Volumes. We assess the diversity of each method using the multi-scale structural similarity metric (MS-SSIM) [30] . MS-SSIM is computed by averaging the results of 400 synthetic sample pairs within each method, serving as a representation of the MS-SSIM within the internal synthesis set. Higher MS-SSIM scores imply that the synthetic volumes generated by a method are more alike, whereas lower scores signify increased diversity. As illustrated in Table 1 , HA-GAN encounter mode collapse, achieving super high MS-SSIM score in dealing with abdominal CT volumes. On the contrary, Medical Diffusion ensures diversity while maintaining generation quality. Lad attains the lowest MS-SSIM score across both datasets, indicating its capability to produce a broader range of samples that faithfully represent the original data distribution. In addition to enhancing the quality of synthetic samples, Lad simultaneously achieves an increase in diversity.Qualitative Comparison of Anatomical Structures. To qualitatively assess the diversity and authenticity of synthetic volumes, Figure 5 presents synthetic samples from each method along with zoomed-in regions on both the AbdomenCT-1K and TotalSegmentator datasets. HA-GAN appears incapable of generating even rough outlines of the abdomen. While Medical Diffusion succeeds in generating complete abdominal structures, the synthesized anatomical details are ambiguous. In contrast, Lad demonstrates superior performance by generating clearer anatomical structures, resulting in more realistic abdominal details.Qualitative Comparison of Synthetic Data Distribution. We embed both synthetic and real volumes into a latent space to assess the degree of overlap in their data distributions. Following the method outlined in [3, 29] , we utilize a pre-trained 3D medical ResNet model [17] to extract features from 800 real and synthetic data samples. Subsequently, Multidimensional Scaling (MDS) [2] is employed to map the extracted features into a 2-dimensional space for both the AbdomenCT-1K and TotalSegmentator datasets. For each method, we fit an ellipse to the embedding with the least squares. From Figure 4 , it is evident that the data distribution of synthetic volumes generated by Lad exhibits the highest degree of overlap with real data. This observation suggests that Lad generates volumes with a more realistic appearance compared to other methods.To assess the effectiveness of synthetic data in self-supervised learning (SSL) tasks, we sample 800 synthetic data from both Medical Diffusion [14] and Lad, which are then utilized as the training set for training the self-supervised segmentation model SSL-ALPNet [22] .For each training set, we conduct five training runs of the segmentation model and evaluate its performance, taking the average of the test results. The dice scores obtained in the segmentation tests of models trained on different training sets are presented in Figure 6 . four abdominal organs. As evidenced by higher dice scores for small organs such as the pancreas and spleen, exceeding Medical Diffusion by 0.9% and 2.7% respectively, our method's emphasis on granularity significantly enhances the delineation of anatomical structures' details. Despite prioritizing local features, our approach achieves comparable performance to Medical Diffusion on larger organs such as the liver and kidney.4.3.2 Synthetic Data for Augmentation. We incorporate synthetic volumes, which are approximately 20% the size of the training set, as a form of data augmentation within the self-supervised segmentation model. Figure 6b illustrates that our approach yields the highest mean dice scores, thereby facilitating effective learning of visual representations by the SSL model. Notably, our method demonstrates particular strength in addressing the challenges associated with small organs, effectively bridging the performance gap observed when compared to Medical Diffusion. 𝐸 𝑠𝑡𝑟 . Initially, we conduct ablation studies on synthetic volumes conditioned on augmented mask sets to discern the individual impact of each component on synthesis quality. Furthermore, to gauge the scalability and adaptability of our method in generating diverse volumes, we extend our analysis to include ablation studies on synthetic volumes conditioned on augmented mask sets without original masks. This scenario simulates encounters with a wide array of "unseen" masks. Table 2 presents a quantitative assessment of synthesized samples across various iterations of our method.Locality Attention Enhances Holistic Quality. A comparison between versions V3 and the Full version, as presented in Table 2 , underscores the significant enhancement in synthesized sample quality upon the introduction of Locality Loss L 𝑙𝑜𝑐 . The samples generated by the Full version consistently achieve the highest scores in terms of both realism and diversity, regardless of the presence of original masks. These findings underscore the pivotal role of our attention mechanism towards locality during the generation process. This attention mechanism enriches synthesized samples with intricate anatomical structure details, ultimately elevating the overall quality of abdominal CT volumes.A comparative analysis among versions V0, V1, and V2 in Table 2 with original masks reveals the importance of incorporating condition guidance in producing volumes that closely resemble real counterparts, as evidenced by the drop in all metrics. Furthermore, comparing versions V1, V2, and V3, it becomes evident that synthesizing samples with superior scores across all metrics is achievable only when both the Content Extractor 𝐸 𝑐𝑜𝑛 and Structure Extractor 𝐸 𝑠𝑡𝑟 are introduced concurrently. This observation underscores the effectiveness of mutually complementary content conditions (𝑐 𝑖 𝑐𝑜𝑛 ) and structure conditions (𝑐 𝑖 𝑠𝑡𝑟 ). However, upon analyzing the results of ablation studies conducted without original masks, it is intriguing to note that V1, featuring the Content Extractor (𝐸 𝑐𝑜𝑛 ) alone, outperforms V3 in terms of MS-SSIM. This observation is reasonable, as the condition in V3 exerts a more potent control over locality details, potentially leading to decreased diversity, which is a trade-off phenomenon. Moreover, the slightly higher MS-SSIM observed in V2 compared to other versions can be attributed to the similarity in topological structures across different volumes, which imposes a less stringent constraint on volume generation.Study. The guidance strength 𝑤 in Eq (4) during sampling represents a trade-off between quality and diversity. We sample with 256 augmented masks for different values of 𝑤, specifically 0.5, 1.0, 1.5, and 2.0, and subsequently conduct a quantitative evaluation on synthetic data. Table 3 demonstrates that synthetic volumes achieve outstanding performance in metrics of both realism and diversity only when 𝑤 = 1.0. 
conclusion: 
 We introduce Locality-Aware Diffusion (Lad), a pioneering method designed specifically for the precise generation of 3D abdominal CT volumes. With a focus on capturing intricate anatomical structure details, we leverage prior knowledge from a well-trained segmentation model, UniverSeg. Through the incorporation of locality refinement, locality condition extraction, and locality condition augmentation modules, we significantly enhance the overall quality of the generated volumes by directing attention to finer details. Experimental results demonstrate that the synthetic abdominal CT volumes produced by our method exhibit realism and diversity across various metrics. These findings underscore the efficacy of synthetic data in facilitating self-supervised tasks. Our approach not only advances the state-of-the-art in abdominal CT volume generation but also opens up new avenues for leveraging synthetic data in medical imaging research. 
 
 Appendix:  


Decision: After rebuttal, the reviewers generally agreed on the contributions of this paper, i.e. domain generalization using synthetic data for medical image analysis. The authors should add additional visualization results on spatial consistency in the final version.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel method for generating high-fidelity 3D abdominal CT volumes using a locality-aware diffusion model (Lad). The method addresses the challenge of generating detailed and realistic anatomical structures in abdominal CT images, which is critical for improving the performance of self-supervised learning (SSL) in medical image analysis. The proposed approach leverages a locality loss and a condition extractor to focus on refining crucial anatomical regions, enabling the generation of high-quality synthetic data that significantly enhances SSL tasks.
strength_weakness: strength_weakness: Novelty: The introduction of the locality-aware diffusion model (Lad) represents a significant advancement in the generation of 3D abdominal CT volumes. The focus on locality details during the generation process is a novel approach that addresses the specific challenges of generating realistic and detailed anatomical structures in abdominal CT images.

Theoretical Approach and Technical Correctness: The paper provides a well-structured and theoretically sound methodology, including the use of VQ-GAN for latent space construction and a locality condition extractor to integrate anatomical priors into the generation process. The inclusion of locality refinement and locality condition augmentation modules further enhances the model's ability to generate high-fidelity synthetic data.

Adequate Evaluation: The extensive experiments and quantitative evaluations demonstrate the effectiveness of the proposed method. The results show significant improvements in metrics such as Fréchet Inception Distance (FID), Maximum Mean Discrepancy (MMD), and multi-scale structural similarity metric (MS-SSIM), indicating the superior quality and diversity of the generated volumes.

Clarity: The paper is well-written and clearly explains the methodology, experimental setup, and results. The use of figures and tables to illustrate the qualitative and quantitative comparisons of the generated volumes with real data and other methods enhances the clarity of the presentation.

Applications: The proposed method has significant implications for self-supervised learning in medical image analysis. The ability to generate high-quality synthetic data without the need for additional labels or radiology reports can reduce the reliance on expensive and time-consuming manual data annotation, thus advancing the field of medical imaging.
limitations: limitations: Complexity of Method: The proposed method involves several components, including VQ-GAN, locality loss, condition extractor, and augmentation modules, which may increase the complexity of implementation and require substantial computational resources.

Generality Across Modalities: While the method shows promising results for abdominal CT volumes, its effectiveness across other anatomical regions and imaging modalities needs further validation. The specificity of the locality-aware approach to the abdomen may limit its generalizability to other medical imaging tasks.

Dependency on Pre-trained Models: The method relies on the use of pre-trained segmentation models (e.g., UniverSeg) for generating anatomical priors. The performance of the proposed method may be influenced by the accuracy and robustness of these pre-trained models.
paper_summary: paper_summary: This manuscript presents a novel method called Locality-Aware Diffusion (Lad), tailored for generating high-quality 3D abdominal CT volumes. It addresses the challenges posed by the intricate anatomical structures of the abdomen, which are often poorly represented in synthetic data. The method integrates a locality loss and a condition extractor to enhance the quality and accuracy of generated volumes. Notably, it achieves significant improvements in FID scores, MMD and MS-SSIM on self-supervised organ segmentation tasks, demonstrating the utility of synthetic data in medical image analysis.
strength_weakness: strength_weakness: 1.	Innovative Approach: This manuscript introduces an innovative blend of locality-aware techniques to improve the synthetic 3D abdominal CT volumes, which is a significant step forward in the field of medical imaging.
2.	Significant Empirical Improvements: The method substantially improves FID, MMD, and MS-SSIM scores, demonstrating its effectiveness over existing methods.
3.	Detailed Methodological Framework: This manuscript presents a well-structured and comprehensive methodological framework, introducing the use of locality loss and condition extractors that integrate anatomical priors effectively.
limitations: limitations: 1.	The authors choose the MS-SSIM to evaluate the diversity of synthetic data. However, MS-SSIM is commonly used to calculate the structural similarity between generated results and a target image when the target is available. The use of this metric in the paper is inappropriate, and the authors need to reconsider its application.
2.	In Figure 4, though the legend indicates that the figure includes four types of data, the absence of the orange ellipse representing HA-GAN calls into question the authenticity of the conclusions.
3.	The visualization results in Figure 5 are visualized using different slices from a single synthetic volume in each of two datasets, which does not demonstrate the diversity of the method proposed in this manuscript.
4.	In Figure 6, the segmentation model trained on the Lad-generated training set shows weaker performance in liver organ segmentation compared to Medical Diffusion, and the paper lacks an explanation for this phenomenon, raising doubts about the rationality and effectiveness of the methods discussed.
5.	In figure 6, there is a lack of evaluation for the segmentation model trained on data synthesized by HA-GAN.
6.	The method in this paper combines various complex models, including VQ-GAN, UniverSeg, DDPM, etc. Does the increased complexity of the models lead to increase computational demands, thereby limiting their usability?
paper_summary: paper_summary: To address the generation of some subtle and complex anatomical structures in abdominal CT scans, this paper introduces Locality-Aware Diffusion (Lad), which is claimed to be the first method specifically tailored for exquisite 3D abdominal CT volume generation. The paper designs a locality loss to refine key anatomical regions and devises a condition extractor to integrate abdominal priors into the generation process, enabling the generation of a large number of high-quality abdominal CT volumes essential for SSL tasks without the need for additional data, such as labels or radiology reports.
strength_weakness: strength_weakness: 1. This paper claims to be the first method specifically tailored for exquisite 3D abdominal CT volume generation.

2. The introduction of the motivation is very clear, and the overall method is also well-written and easy to understand.

3. The ablation study is quite comprehensive, with an ablation of each module.
limitations: limitations: 1. Are the contents from pages 320-326 repetitive of those from pages 283-289? They seem to convey almost the same meaning.

2. For both comparative experiments and ablation studies, it is best to display the segmentation results to make them clearer, as shown in Figure 1(b).

3. How does this paper maintain spatial consistency between slices for the 3D generation method?