SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} u \sim U n i f o r m [ 0, 1 ] \\ (z _ {0}, c) = \left\{ \begin{array}{l l} (z _ {0}, \emptyset) \sim q _ {0} (z _ {0}) & \text {i f u \geqslant p _ {u n c}} \\ (z _ {0}, c) \sim q _ {1} (z _ {0}, c) & \text {o t h e r w i s e} \end{array} \right. \\ t \sim U n i f o r m \{1, \dots \\ 3: \text {S a m p l e n o i s e ,} \epsilon \sim \mathcal {N} (0, \mathbf {I}) \\ 4: \text {C o r r u p t} z _ {t} = \gamma_ {t} x _ {0} + \sigma_ {t} e \\ 5: \text {T a k e g r a i d e n t} \\ \nabla_ {\theta} \| \epsilon - \epsilon_ {\theta} (z _ {t}, t, c) \| ^ {2} \\ 6: \text {u n t i l} \\ \end{array}
$$
Algorithm 2 DiffInfinite Sampling
Input: High-level segmentation mask  $M\in \mathbb{R}^{D}$  and learned model  $\epsilon_{\theta}$
Output: Synthetic image  $X$  with the mask size
Initialization:
$\mathbf{y}_T \sim \mathcal{N}(0, \mathbf{I})$ , index set  $I_0 = \varnothing$  and time state tensor  $L_T \equiv T$
1: for  $t \in \{T - 1, \dots, 0\}$  do
2: while  $\cup_{j=0}^{m} I_{j} \neq \{1, \dots, D\}$  do
3:  $m \gets m + 1$
4: Select randomly  $proj_{I_m} \in \mathcal{C} \backslash \{proj_{I_1}, \dots, proj_{I_{m-1}}\}$
5: Crop  $z_{t}^{m} = \text{proj}_{I_{m}}(y_{t})$
6: for all conditions  $n \in \{1, \dots, N\}$  do
7: DDIM sampling with classifier-free guidance
$$
z _ {t - 1} ^ {m} \mid c _ {n} \sim p _ {\theta , t} (z \mid z _ {t} ^ {m}, c _ {n})
$$
10: for all indices  $i\in I_m$  do
11: if  $i \notin I_j$  for all  $j < m$  then
12:  $[y_{t - 1}]_i\gets [z_{t - 1}^m |M_i]_l$
13:  $\text{proj}_{I_m}(L_t)_i \gets t$
14: such that  $[proj_{I_m}(y_{t - 1})]_l = [z_{t - 1}^m |M_i]_l$
15: end if
16: end for
17: end while
18: end for
19:  $X\gets \Psi (y_0)$
5 Data Assessment
To assess synthetic images for medical image analysis, we need to take various dimensions of data assessment into account. We extend traditional metrics from the natural image community with qualitative and quantitative assessments specific to the medical context. For the qualitative analysis, a team of pathologists evaluated the images for histological plausibility. The quantitative assessment entailed a proof-of-concept that a model can learn sensible features from the synthetically generated image patches for a relevant downstream task. As data protection is highly relevant regarding patient data, we performed evaluations to rule out memorization effects of the generative model.
Table 1: Metrics to quantitatively evaluate the quality of the generated images. Left: scores for images of size  $512 \times 512$ . DiffInfinite (a) first generates a mask and secondly an image following Section 4.1. Right: scores for real and generated images of size  $2048 \times 2048$  resized to  $512 \times 512$ . All methods use the same model trained on small patches of size  $512 \times 512$ . DiffCollage corresponds to the method proposed in [48]. DiffInfinite (b) uses the real masks, while DiffInfinite (c) first generates a mask and secondly the large image. DiffInfinite (b) & (c) refers to the mixture of the generated dataset from DiffInfinite (b) and DiffInfinite (c).
IP↑	IR↑	IS↑	FID↓	IP↑	IR↑
Morph-Diffusion [72]	0.26	0.85	2.1	20.1	DiffCollage	0.94	0.22
NASDAQ [73]	-	-	2.7	15.7	DiffInfinite (b)	0.95	0.48
DiffInfinite (a)	0.94	0.70	2.7	26.7	DiffInfinite (c)	0.98	0.44
DiffInfinite (b) & (c)	0.98	0.33
5.1 Traditional Fidelity
We evaluate the fidelity of synthetic  $512 \times 512$  images by calculating Improved Precision (IP) and Improved Recall (IR) metrics between 10240 real and synthetic images [67]. The IP evaluates synthetic data quality, while the IR measures data coverage. Despite their unsuitability for histological data [68, 69], Frechet-Inception Distance (FID) and Inception Score (IS) [70, 71] are reported for comparison with [72] and Shrivastava and Fletcher [73]. The metrics' explanations and formulas can be found in Appendix C.
In Table 1 (left), we report an IP of 0.94 and an IR of 0.70, indicating good quality and coverage of the generated samples. However, we note that these metrics are only somewhat comparable due to the different types of images generated by MorphDiffusion [72] and NASDM [73]. For the large images of size  $2048 \times 2048$ , we rely solely on the IP and IR for quantitative evaluation due to the limited number of 200 generated large images. As shown in Figure 3(a) of [67], FID is unsuitable for evaluating such a small sample size, while IP and IR are more reliable. In Table 1 (right), we find that generating images first results in slightly higher IR, while generating the mask first achieves an IP of 0.98. For the sake of completeness we also report the scores then combining the two datasets. To compare our method to DiffCollage we generate 200 images using [48]. DiffInfinite performs better than DiffCollage wrt. to IP and IR. The drop of IR to 0.22 might be a result of the tiling artifacts observable in the LHS of Figure 11.
5.2 Domain Experts Assessment
To assess the histological plausibility of our generated images, we conducted a survey with a cohort of ten experienced pathologists, averaging 8.7 years of professional tenure. The pathologists were tasked with differentiating between our synthesized images and real image patches extracted from whole slide images. We included both small patches  $(512 \times 512\mathrm{px})$  as commonly used for downstream tasks as well as large patches  $(2048 \times 2048\mathrm{px})$ . Including large patches enabled us to additionally evaluate the modelled long-range correlations in terms of transitions between tissue types as well as growth patterns which are usually not observable on the smaller patch sizes but essential in histopathology. In total the survey contained 60 images, in equal parts synthetic and real images as well as small and large patches. The overall ability of pathologists to discern between real and synthetic images was modest, with an accuracy of  $63\%$ , and an average reported confidence level of 2.22 on a 1-7 Likert scale. While we observed high inter-rater variance, there was no clear correlation between experience and accuracy  $(r(8) = .069$ ,  $p = .850)$ , nor between confidence level and accuracy  $(r(8) = .446$ ,  $p = .197)$ . Furthermore there was no significant correlation between the participants' completion time of the survey and the number of correct responses  $(r(8) = -.08$ ,  $p = .826)$ .
Surprisingly, we found a similar performance for both, real and synthetic images. This indicates that, while clinical practice is mostly based on visual assessment, it is not a common task for pathologists to be restricted to parts of the whole slide image only. More detailed visualizations of the individual scores can be found in Appendix B. Besides this satisfactory result, we additionally wanted to
$^{2}$ https://github.com/blandocs/improved-precision-and-recall-metric-pytorch
<sup>3</sup>https://github.com/toshas/torch-fidelity
Table 2: Zero-shot evaluation results of the downstream tasks, encompassing both classification and segmentation scenarios. We employed three distinct models for each scenario: The first, "Trained Real," was trained using real data (in-house IH1), which also served as the training set for DiffInfinite. The second, "Trained Synthetic," was trained using samples generated from DiffInfinite, and the third, "Trained Augmented," utilized a combination of real and synthetic data. Our evaluation extends across separate lung cohorts (internal datasets IH2 and IH3) and additional indications (external datasets NCT, CRC, PCam), with varying degrees of data drift introduced.
IH1	IH2	IH3	NCT-100K	CRC-7K	PCam-327K
Drift components	-	Patient change Different center	Patient change Different center Indication change	Patient change Different center Indication change Lower resolution
Trained Real	0.846 ± 0.005	0.733 ± 0.021	0.598 ± 0.049	0.857 ± 0.009	0.822 ± 0.034	0.628 ± 0.035
Trained Synthetic	0.747 ± 0.025	0.753 ± 0.005	0.699 ± 0.002	0.796 ± 0.023	0.753 ± 0.038	0.628 ± 0.012
Trained Augmented	0.852 ± 0.007	0.732 ± 0.027	0.637 ± 0.025	0.847 ± 0.044	0.811 ± 0.057	0.641 ± 0.035
(a) Classification results
IH2
Trained Real	0.614 ± 0.009
Trained Synthetic	0.471 ± 0.039
Trained Augmented	0.710 ± 0.021
(b) Segmentation results
explore the limitations of our method by assessing the nuanced differences pathologists observed between synthetic and real images. While overall the structure and features seemed similar and hard to discern, they sometimes reported regions of inconsistent patterns, overly homogeneous chromatin in some of the synthetic nuclei, peculiarities in cellular and intercellular structures, and aesthetic elements. These seemed to be especially pronounced in tumorous regions where sometimes the tissue architecture appeared exaggerated, the transition to stroma or surrounding tissue was too abrupt and some cells lacked distinguishable nucleoli or cytoplasm. We attribute the nuanced effect of larger image size on the accuracy on this observation (cf. Fig. 5C). Overall the finding of the conducted survey demonstrates how complex the task of distinguishing between real and synthetically generated data is even for experienced pathologists while still highlighting potential areas to improve the generative model.
5.3 Synthetic Data for Downstream Tasks
A major interest in the availability of high quality labeled synthetic images is their use in downstream digital pathology applications. In this area, two primary challenges are the binary classification of images into cancerous or healthy tissues and the segmentation of distinct tissue areas in the tumor microenvironment. The unique ability of our technique to generate images of different cancer subtypes through the context prompt as well as the ability to create new segmentation masks and their corresponding H&E images specifically addresses these two challenges. Notably, expert annotations are costly and time consuming to acquire thus emphasizing the benefits of being able to train on purely synthetic datasets or augmenting annotated data in the low data regime. To showcase these two usecases we performed a series of experiments in both classification and segmentation settings. For all experiments, we trained a baseline classifier on a relatively small number of expert annotations IH1 (#patches = 3726)) — the same that were used to train DiffInfinite — and additionally trained one model purely on synthetic data (IH1-S, #patches = 9974,  $\omega = 0$ ), and one model on the real data augmented with the synthetic images. To generate target labels for the classification experiments, we simplified the segmentation challenge by categorizing patches with at least  $0.05\%$  of pixels labeled as 'Carcinoma' in the segmentation masks as 'Carcinoma'. All other patches were labeled 'Non-Carcinoma'. We evaluated all three classification models on several out-of-distribution datasets. We utilized two proprietary datasets (from the same cancer type with similar attributes but from distinct patient groups: IH1 (# patients=13, # patches=704) and IH3 (# patients=2, # patches=2817). Moreover, we assessed the models using two public datasets (NCK-CRC [74] and PatchCamelyon [75]), both representing tissue from different organs with distinct morphologies. Our findings, summarized in Table 2a, suggest that a classifier's out-of-distribution performance, trained with limited sample size and morphological diversity, can vary significantly (ranging from 0.628 to 0.857 balanced accuracy). This variability cannot be attributed solely to morphology but may also be influenced by
factors such as resolution and variations in scanning and staining techniques. Training exclusively with a larger set of synthetic images can enhance performance on some datasets (specifically IH2 and IH3), underscoring the advantages of leveraging the full training data in a semi-supervised manner within the generative model. Incorporating synthetic data as an augmentation to real data not only prevents the classifier's performance decline, as seen on NCT-CRC and Patchcamelyon, on similar datasets but also bolsters its efficiency on more distinct ones. For the more challenging segmentation task we again trained three segmentation models to differentiate between carcinoma, stroma, necrosis, and a miscellaneous class that included all other tissue types, such as artifacts. The baseline performance of the real data model on a distinct group of lung patients (dataset IH2) of a  $F_{1}$  score of  $0.614 \pm 0.009$  (across three random seeds) highlights the difficulty of generalizing out of distribution in this tasks. While the purely synthetic model was not able to fully recover the baseline performance  $(0.471 \pm 0.039)$ , augmenting the small annotated dataset with synthetic data enhanced predictive performance to an  $F_{1}$  score of  $0.710 \pm 0.021$ . This boost of 10 percentage points in performance demonstrates that the synthetic data provide new, relevant information to the downstream task. In summary, our findings demonstrate the feasibility of meeting or surpassing baseline performance levels for both tasks using either entirely synthetic data or within an augmented context. Nevertheless, the advantages of employing synthetic data in downstream tasks continue to pose a challenge, not only within the medical image domain but also across various other domains [76-78], thus requiring more comprehensive assessment and thorough examination.
5.4 Considerations on Memorization
In medicine the adherence to privacy regulations is a sensitive requirement. While it is generally not possible for domain experts to infer patient identities from the image content of a histological tile or slide alone [79], developers and users of generative models are well advised to understand the risk of correspondence between the training data and the synthesized data. To this end, we evaluate the training and synthesized data against two memorization measures. The authenticity score  $A \in [0,1]$  by [54] aims to measure the rate by which a model generates new samples (higher score means more innovative samples). Similarly, [80] aims to estimate the degree of data copying  $C_T$  from the training data by the generative model. A  $C_T \ll 0$  implies data copying, while a  $C_T \gg 0$  implies an underfitting of the model. The closer to 0 the better. See Appendix C for a precise closed form of the measures and Table 5 for the full quantitative results, indicating that the DiffInfinite model is not prone to data copying across all resolutions and variations considered here<sup>4</sup>. The  $A$  range between 0.86 and 0.98, signifying a high rate of authenticity. While other papers unfortunately do not report such detailed memorization statistics for their models, the results by [54] suggest that a score  $\gg 0.8$  is not trivial to achieve. None of the models under consideration in [54] (VAE, DCGAN, WGAN-GP, ADS-GAN) achieve more than 0.82 in  $A$  on simpler data (MNIST). This interpretation is strengthened by the results of a  $C_T \gg 0$  which indicates that the model might even be underfitting and is not in a data copying regime. Qualitative results on the nearest neighbour search between training and synthetic data in Figure 1 further corroborate these quantitative results.
DiffInfinite offers a novel sampling method to generate large images in digital pathology. Due to the high-level mask generation followed by the low-level image generation, synthetic images contain long-range correlations while maintaining high-quality details. Since the model trains and samples on small patches, it can be efficiently parallelized. We demonstrated that the classifier-free guidance can be extended to a semi-supervised learning method, expanding the labelled data feature space with unlabelled data. The biological plausibility of the synthetic images was assessed in a survey by 10 domain experts. Despite their training, most participants found it challenging to differentiate between real and synthetic data, reporting an average low confidence in their decisions. We found that samples from DiffInfinite can help in certain downstream machine learning tasks, on both in- as well as out-of-distribution datasets. Finally, authenticity metrics validate DiffInfinite's capacity to generate novel data points with little similarity to the training data which is beneficial for the privacy preserving use of generative models in medicine.
<sup>4</sup>We use https://github.com/marcojira/cls from [81] to calculate both scores.
We would like to acknowledge our team of pathologists who provided valuable feedback in and outside of the conducted survey - special thank you to Frank Dubois, Niklas Prenissl, Cleopatra Schreiber, Vitaly Garg, Alexander Arnold, Sonia Villegas, Rosemarie Krupar and Simon Schallenberg. Furthermore, we would like to thank Marvin Sextro for his support in the analyses. This work was supported by the Federal Ministry of Education and Research (BMBF) as grants [SyReal (01IS21069B)]. RM-S is grateful for EPSRC support through grants EP/T00097X/1, EP/R018634/1 and EP/T021020/1, and DI for EP/R513222/1. MA is funded by Dotphoton, QuantIC and a UofG Ph.D. scholarship.
[1] Ravi Aggarwal, Viknesh Sounderajah, Guy Martin, Daniel SW Ting, Alan Karthikesalingam, Dominic King, Hutan Ashrafian, and Ara Darzi. Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ digital medicine, 4(1):65, 2021.
[2] Antonio Parziale, Monica Agrawal, Shengpu Tang, Kristen Severson, Luis Oala, Adarsh Subbaswamy, Sayantan Kumar, Elora Schoerverth, Stefan Hegselmann, Helen Zhou, Ghada Zamzmi, Purity Mugambi, Elena Sizikova, Girmaw Abebe Tadesse, Yuyin Zhou, Taylor Killian, Haoran Zhang, Fahad Kamran, Andrea Hobby, Mars Huang, Ahmed Alaa, Harvineet Singh, Irene Y. Chen, and Shalmali Joshi. Machine learning for health (ml4h) 2022. In Antonio Parziale, Monica Agrawal, Shalmali Joshi, Irene Y. Chen, Shengpu Tang, Luis Oala, and Adarsh Subbaswamy, editors, Proceedings of the 2nd Machine Learning for Health symposium, volume 193 of Proceedings of Machine Learning Research, pages 1-11. PMLR, 28 Nov 2022.
[3] Maximilian Springenberg, Annika Frommholz, Markus Wenzel, Eva Weicken, Jackie Ma, and Nils Strodthoff. From modern cnns to vision transformers: Assessing the performance, robustness, and classification strategies of deep learning models in histopathology. Medical Image Analysis, 87:102809, 2023. ISSN 1361-8415.
[4] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284, 2023.
[5] Miriam Hägele, Philipp Seegerer, Sebastian Lapuschkin, Michael Bockmayr, Wojciech Samek, Frederick Klauschen, Klaus-Robert Müller, and Alexander Binder. Resolving challenges in deep learning-based analyses of histopathological images using explanation methods. *Scientific reports*, 10(1):1–12, 2020.
[6] Jun Ma and Bo Wang. Segment anything in medical images. arXiv preprint arXiv:2304.12306, 2023.
[7] Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al. The future of digital health with federated learning. NPJ digital medicine, 3(1):119, 2020.
[8] Luis Oala, Jana Fehr, Luca Gilli, Pradeep Balachandran, Alixandro Werneck Leite, Saul Calderon-Ramirez, Danny Xie Li, Gabriel Nobis, Erick Alejandro Muñoz Alvarado, Giovanna Jaramillo-Gutierrez, Christian Matek, Arun Shroff, Ferath Kherif, Bruno Sanguinetti, and Thomas Wiegand. Ml4h auditing: From paper to practice. In Emily Alsentzer, Matthew B. A. McDermott, Fabian Falck, Suproteem K. Sarkar, Subhrajit Roy, and Stephanie L. Hyland, editors, Proceedings of the Machine Learning for Health NeurIPS Workshop, volume 136 of Proceedings of Machine Learning Research, pages 280-317. PMLR, 11 Dec 2020.
[9] Christopher G Schwarz, Walter K Kremers, Terry M Therneau, Richard R Sharp, Jeffrey L Gunter, Prashanthi Vemuri, Arvin Arani, Anthony J Spychalla, Kejal Kantarci, David S Knopman, et al. Identification of anonymous mri research participants with face-recognition software. New England Journal of Medicine, 381(17):1684-1686, 2019.
[73] Aman Shrivastava and P Thomas Fletcher. Nasdm: Nuclei-aware semantic histopathology image generation using diffusion models. arXiv preprint arXiv:2303.11477, 2023.
[74] Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue, April 2018. URL https://doi.org/10.5281/ zenodo.1214456.
[75] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pages 210-218. Springer, 2018.
[76] Shenghuan Sun, Gregory M Goldgof, Atul Butte, and Ahmed M Alaa. Aligning synthetic medical images with clinical knowledge using human feedback. arXiv preprint arXiv:2306.12438, 2023.
[77] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259, 2023.
[78] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493, 2023.
[79] Petr Holub, Heimo Müller, Tomáš Bíl, Luca Pireddu, Markus Plass, Fabian Prasser, Irene Schlünder, Kurt Zatloukal, Rudolf Nenutil, and Tomáš Brázdil. Privacy risks of whole-slide image sharing in digital pathology. Nature Communications, 14(1):2577, 2023.
[80] Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. A non-parametric test to detect data-copying in generative models. CoRR, abs/2004.05675, 2020.
[81] Marco Jiralerspong, Avishek Joey Bose, Ian Gemp, Chongli Qin, Yoram Bachrach, and Gauthier Gidel. Feature likelihood score: Evaluating generalization of generative models using samples, 2023.
[82] Nicolas Pielawski and Carolina Wahlby. Introducing hann windows for reducing edge-effects in patch-based image segmentation. *PloS one*, 15(3):e0229839, 2020.