Title: Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology
Venue: NeurIPS 2025
Accepted: yes
Keywords: Histopathology

Generative models

Diffusion models

Image synthesis

Heterogeneous tissue

Self-supervised diffusion model
Abstract: Synthetic data generation in histopathology faces unique challenges: preserving tissue heterogeneity, capturing subtle morphological features, and scaling to unannotated datasets. We present a latent diffusion model that generates realistic heterogeneous histopathology images through a novel dual-conditioning approach combining semantic segmentation maps with tissue-specific visual crops. Unlike existing methods that rely on text prompts or abstract visual embeddings, our approach preserves critical morphological details by directly incorporating raw tissue crops from corresponding semantic regions. For annotated datasets (i.e., Camelyon16, Panda), we extract patches ensuring 20-80% tissue heterogeneity. For unannotated data (i.e., TCGA), we introduce a self-supervised extension that clusters whole-slide images into 100 tissue types using foundation model embeddings, automatically generating pseudo-semantic maps for training. Our method synthesizes high-fidelity images with precise region-wise annotations, achieving superior performance on downstream segmentation tasks. When evaluated on annotated datasets, models trained on our synthetic data show competitive performance to those trained on real data, demonstrating the utility of controlled heterogeneous tissue generation. In quantitative evaluation, prompt‐guided synthesis reduces Fréchet Distance by up to 6× on Camelyon16 (from 430.1 to 72.0) and yields 2–3× lower FD across Panda and TCGA. Downstream DeepLabv3+ models trained solely on synthetic data attain test IoU of 0.71 and 0.95 on Camelyon16 and Panda, within 1–2% of real‐data baselines (0.72 and 0.96). By scaling to 11,765 TCGA whole‐slide images without manual annotations, our framework offers a practical solution for an urgent need for generating diverse, annotated histopathology data, addressing a critical bottleneck in computational pathology.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces HeteroTissue-Diffuse (HTD), a latent diffusion model for synthesizing heterogeneous histopathology images. The key innovation is a dual-conditioning mechanism that combines semantic segmentation maps with tissue-specific visual crops, preserving fine-grained morphological details. The method is evaluated on annotated datasets (Camelyon16, PANDA) and extended to unannotated TCGA data via self-supervised clustering of 100 tissue types. Quantitative results show significant improvements in fidelity (Frechet Distance reduced by up to 6×) and downstream task performance (segmentation IoU within 1–2% of real-data baselines). Expert pathologists rated synthetic images as indistinguishable from real ones. The work addresses data scarcity and privacy concerns in computational pathology.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1) Dual-Conditioning Approach that combines semantic maps with raw visual crops, avoiding information loss in text/embedding-based methods and preserves diagnostically critical features (nuclear texture, staining patterns) better than prior work.\n\n2) Scalability to Unannotated Data: Self-supervised clustering on TCGA (11,765 WSIs) enables large-scale synthetic data generation without manual labels. Lightweight classifier during inference reduces computational costs by 85%\n\n3) Quantitative metrics (Frechet Distance, IoU) and blinded pathologist evaluations demonstrate clinical realism. Synthetic data nearly matches real data in segmentation performance (IoU 0.71 vs. 0.72 on Camelyon16)\n\n4) Code, pre-trained models, and the generated synthetic datasets will be made available \n\nWeaknesses:\n\n1) Computational Cost: Training requires 4 months on 4× NVIDIA A100 GPUs; clustering TCGA took 48 hours on a 124-core server. 3 months to extract the entire TCGA 224x224 patch embeddings of the high magnification of each 197 WSI on 1 x NVIDIA A100 GPUs with 80GB This may limit accessibility for resource-constrained institutions. \n\n2) Rare Pattern Coverage: Clustering into 100 tissue types may miss extremely rare pathologies (<0.1% prevalence), potentially affecting niche diagnostic tasks.\n\n3) Somewhat limited novelty: existing latent based (semantic mask conditioned) diffusion model applied on large scale data.\n\n4) Downstream tasks focus only on segmentation; diagnostic accuracy (e.g., tumor grading) is not tested.\n\nQUESTIONS:\n1) Does the model maintain structural coherence across large WSIs, or are there artifacts at slide-level scales?\n\n2) Why did you train UNet instead of DiT? Isn't DiT better for large-scale datasets?\n\n3) Could the framework be adapted for non-H&E stains or multi-modal data (e.g., paired H&E/IHC (immunochemistry) slides)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of generating realistic synthetic histopathology images that preserve tissue heterogeneity and fine morphological detail. Traditional generative approaches in this domain have struggled with controlling multiple tissue types and often rely on text prompts or compressed embeddings, which may lose critical diagnostic features. This work proposes a latent diffusion model that uses a novel dual-conditioning strategy which conditions generation on both a semantic segmentation map and raw tissue image crops from each region. By directly incorporating actual image patches as exemplars, the model preserves subtle histological details like nuclear texture and staining patterns that might be lost with text or abstract embeddings. Furthermore, to handle unlabeled data, the authors introduce a self-supervised pipeline which clusters slides into 100 pseudo tissue types and generate semantic masks without manual annotation. Evaluation results demonstrate that the approach can synthesize high-fidelity, multi-tissue pathology images with accurate region annotations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed diffusion framework directly tackles limitations of prior models by preserving fine-grained pathology details via crop-based conditioning, which is interesting, and shows clear improvements in image realism and downstream task performance.\n\n2. The author conducts solid measurement, e.g., clinical plausibility is corroborated by a blinded study with certified pathologist., distributional realism is measured with eight foundation-model encoders, downstream utility is assessed through end-to-end training of segmentation networks.\n\n3. The proposed model's ability to synthesize region-annotated, multi-tissue patches that nearly match real data in segmentation performance addresses a critical bottleneck in computational pathology.\n\nWeaknesses:\n1. As mentioned by the authors, the proposed method requires several months of A100-level GPU time plus a large CPU cluster for embedding extraction and clustering, which is a substantial computational cost.\n\n2. Stain and modality generalization is untested. All experiments use H&E staining, it is unknown whether the proposed method can apply to other staining like immunohistochemistry.\n\n3. Unquantified cluster quality. The self-supervised clustering of 564 M patches into 100 tissue should be explained or analyzed, for example, bad cases or examples or quantified nums (will be better) should be provided to prove the cluster quality.\n\nQUESTIONS:\n1. All experiments in the paper mainly focus on H&E slides. Do you anticipate that the dual-conditioning mechanism will transfer to other modalities such as IHC, special stains?\n\n2. The self-supervised clustering step is critical yet its quality is not rigorously analyzed. Could you report quantitative measures of cluster purity or conduct some analysis? And why choose 100 clusters?\n\n3. The generation relies on supplying tissue-specific visual crops as exemplars. In practice, how are these exemplars chosen for new, unseen regions especially if a rare tissue type lacks a high-quality crop? Would an averaged prototype, or does generation quality degrade sharply without carefully curated exemplars?\n\nLIMITATIONS:\nPlease kindly refer to the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes HeteroTissue-Diffuse (HTD) - a latent diffusion model for generating synthetic histopathology patches - in which the generative process is conditioned on both a semantic map and a visual tissue prompt. The generated data is evaluated along three axes: quantitatively via the Fréchet Distance; by the IoU of a tissue segmentation model trained solely on synthetic data; and qualitatively through assessment by a pathologist. Moreover, the paper provides a clustering for the TCGA dataset into 100 distinct tissue phenotype without manual annotation. The authors commit to open-sourcing their code upon publication.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper is well written and clearly structured.\n- The generated data is thoroughly evaluated quantitatively across three different datasets and eight foundation model encoders.\n- In addition to the quantitative evaluation, the generated data is assessed by a pathologist with seven years of clinical experience in surgical pathology, with the synthetic data achieving higher scores in this assessment for overall quality, structural detail, and nuclear detail across the CAMELYON, PANDA, and TCGA datasets.\n- Notably, the generated synthetic data achieves nearly identical performance on the downstream tissue segmentation task, as measured by test IoU.\n- The authors commit to open-sourcing their code upon publication.\n\n### Weaknesses\n- The authors repeatedly claim that HTD preserves patient privacy (e.g., lines 71, 138, 282, 287, 345, 457); however, there is no attempt to qualitatively or quantitatively assess whether the generated data is indeed privacy-preserving beyond the fact that it is synthetic. The reviewer acknowledges that this question cannot be answered with complete certainty; however, there should be some discussion of this topic in the manuscript.\n- The authors state in lines 261 to 263: “The results demonstrate that prompt conditioning significantly improves generation quality compared to the non-prompt (NP) baseline across all datasets,” referring to the FD scores reported in Table 1. However, on the TCGA dataset, the two best FD scores—117.5 and 119.6—are achieved with the NP baseline, and on the CAM16 dataset, the second-best FD score of 70.0 is also achieved with the NP baseline. The reviewer therefore suggests a more cautious formulation, for example: “The results demonstrate that prompt conditioning improves generation quality on the PANDA and CAM16 datasets, whereas on the TCGA dataset, the overall best performance is achieved by the NP baseline.\"\n\nQUESTIONS:\nThe paper is missing an adequate discussion on the privacy of the generated data. To improve the quality of the paper and address the weaknesses outlined above, the reviewer suggests the following steps:\n- Add a quantitative evaluation of the generated data’s privacy, using, for example, the Authenticity score proposed in Alaa et al. [1] or the $C_T$ score proposed in Meehan et al. [2]. Both metrics are implemented in the repository of Jiralerspong et al. [3] (https://github.com/marcojira/FLD). This suggestion is not exhaustive, and the authors are welcome to use other appropriate metrics to assess the privacy of the generated data.\n- Include a qualitative comparison of training and generated data. For instance, the authors could follow the approach of Aversa et al. [4], in which generated images are visually compared to the closest real images identified using an Inception-v3 nearest-neighbor search. This is merely one possible strategy; the authors are encouraged to adopt or propose an alternative method for qualitatively evaluating the privacy of the generated data.\n- Revise lines 261 to 263 to adopt a more cautious formulation that accurately reflects the scores reported in Table 1.\n\n### Minor comments:\n- (line 132): missing reference\n- No valid caption on Figure 4 - caption is copied from Figure 3\n\n[1] Alaa et al. How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models. ICML 2022.\n\n[2] Meehan et al. A non-parametric test to detect data-copying in generative models. CoRR, 2020.\n\n[3] Jiralerspong et al. Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples. NeurIPS, 2023. \n\n[4] Aversa et al. DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathology. NeurIPS 2023.\n\nLIMITATIONS:\nThe authors have thoroughly addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a latent diffusion model for generating synthetic histopathology images by conditioning both on segmentation maps and visual tissue crops. The tissue crops help to match general tissue characteristics in a region (texture, cellular morphology, and staining patterns) while the segmentation maps enable spatial control. For datasets without annotations, the images are automatically clustered into 100 tissue types using foundation model embeddings to create pseudo-semantic maps. The authors train on up to 11k WSI images from TCGA and show improved FID over other conditioning mechanisms. Further, they compare downstream segmentation models trained on synthetic data with models trained on real data almost matching the performance of real-data baselines with synthetic data (within 1–2 %).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper is well written and puts multiple concepts nicely together.\n* The resulting images are evaluated from multiple angles (FID, topological FID, expert pathologist evaluation, downstream segmentation models).\n* The methods are well explained and figures are clear and supportive of the main content.\n* The 6x fold improvement in FID is impressive.\n\n**Weaknesses**\n\n* The “Downstream Evaluation - Tissue Segmentation” section in the results lacks information on how the synthetic examples were generated. If the respective diffusion models were pretrained on the exact segmentation datasets, I would argue that the diffusion model can just memorize/model the training data well and the matching of real data performance is not surprising. This needs further description on how the diffusion models were trained exactly and which data was left out to be able to understand the impact.\n* There is no evaluation whether a mix of real and synthetic data can improve the downstream task performance (e.g. by systematic generation of different spatial patterns/class combinations). The privacy preserving training is of limited utility from my perspective as also the diffusion model needs to be trained on that data. Boosting the performance through systematic additional training data would make the paper much stronger.\n* The tissue type discovery via deep clustering approach (Section 3.3.1) is presented as novel. However, something really similar has already been explored in [1]. While this work definitely extends the scope of this work and combines it with visual prompts and semantic maps, [1] should be cited and novelty claims in this direction reduced. Further, the Conditioning Mechanisms for Histopathology Synthesis section in Related work claims that there is just no conditioning, text conditioning or based on SSL methods. But there exists also works that e.g. explore further metadata conditioning [2, 3] and conditioning on RNA data [4] for histopathology.\n* Minor: Citation missing in line 132 (?)\n* Minor: H-optimus -> H-optimus-0 and UNI2 -> UNI2-h  (more clear) in Table 1.\n\n[1] Osorio, Pedro et al. “Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology.” Diagnostics (Basel, Switzerland) vol. 14,13 1442. 5 Jul. 2024, doi:10.3390/diagnostics14131442\n\n[2] Ktena, Ira, et al. \"Generative models improve fairness of medical classifiers under distribution shifts.\" Nature Medicine 30.4 (2024): 1166-1173.\n\n[3] Drexlin, David Jacob, et al. \"MeDi: Metadata-Guided Diffusion Models for Mitigating Biases in Tumor Classification.\" arXiv preprint arXiv:2506.17140 (2025).\n\n[4] Carrillo-Perez, Francisco, et al. \"Generation of synthetic whole-slide image tiles of tumours from RNA-sequencing data via cascaded diffusion models.\" Nature Biomedical Engineering 9.3 (2025): 320-332.\n\nQUESTIONS:\n* On which data was the diffusion model trained for the Tissue Segmentation downstream evaluation?\n* How were the visual crops chosen during the inference?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}