DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathology
Authors: Marco Aversa
Gabriel Nobis
Miriam Hägele
Kai Standvoss
Mihaela Chirica
Roderick Murray-Smith
Ahmed Alaa
Lukas Ruff
Daniela Ivanova
Wojciech Samek
Frederick Klauschen
Bruno Sanguinetti
Luis Oala
Venue: NeurIPS 2023 Datasets and Benchmarks Spotlight (2023)
Keywords: diffusion models
generative models
histopathology
medical imaging

Abstract: We present DiffInfinite, a hierarchical diffusion model that generates arbitrarily large histological images while preserving long-range correlation structural information. Our approach first generates synthetic segmentation masks, subsequently used as conditions for the high-fidelity generative diffusion process. The proposed sampling method can be scaled up to any desired image size while only requiring small patches for fast training. Moreover, it can be parallelized more efficiently than previous large-content generation methods while avoiding tiling artifacts. The training leverages classifier-free guidance to augment a small, sparsely annotated dataset with unlabelled data. Our method alleviates unique challenges in histopathological imaging practice: large-scale information, costly manual annotation, and protective data handling. The biological plausibility of DiffInfinite data is evaluated in a survey by ten experienced pathologists as well as a downstream classification and segmentation task. Samples from the model score strongly on anti-copying metrics which is relevant for the protection of patient data.

DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathology
Marco Aversa
Gabriel Nobis
University of Glasgow and Dotphoton
Fraunhofer HHI
Glasgow, United Kingdom
Berlin, Germany
marco aaseo.alls.aaaaag........o.o... uk
gabriel. nobis@hhi fraunhofer de
Miriam Hägele
Kai Standvoss
Aignostics
Aignostics
Berlin, Germany
Berlin, Germany
miriam hww.:heeeelliiggiiiis....o
kai www.advoossiiiinssssss.ooo com
Mihaela Chirica
Roderick Murray-Smith
Institute of Pathology, LMU Munich
University of Glasgow
Munich, Germany
Glasgow, United Kingdom
mihaela, chiri cammed. uni-muenchen.de
roderick. smderickk
ac. uk
Ahmed Alaa
Lukas Ruff
UC Berkeley Berkeley, California
Aignostics Berlin, Germany
amalaa@berkeley. edu
lukas deitas.ufff@aee wwfffaaignosticcs.com
Daniela Ivanova University of Glasgow Glasgow, United Kingdom
Wojeiech Samek
Fraunhofer HHI and TU Berlin
Berlin, Germany
daniela. ivaiiaaa.vvaooo@glaaggooo ac uk
kojciech.sameeethhi... fraunhofer. de
samek@hhi.........
Frederick Klauschen
Bruno Sanguinetti
Institute of Pathology, LMU Munich Munich, Germany kllaschhen@mm.dee
Dotphoton
Zug, Switzerland
bruno bruno. www..sagguueettilloootottooo...o
Luis Oala
Dotphoton
Zug, Switzerland
luis. wat..aallltpphhoon.ccom
Abstract
We present DiffInfinite, a hierarchical diffusion model that generates arbitrarily large histological images while preserving long-range correlation structural infor- mation. Our approach first generates synthetic segmentation masks, subsequently used as conditions for the high-fidelinn generative diffusion process. The proposed sampling method can be scaled up any desired image size while only requiring
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Bench- marks.

---

small patches for fast training. Moreover, it can be parallelized more efficiently than previous large-content generation methods while avoiding tiling artifacts. The training leverages classifier-free guidance to augment small, sparsely an- notated dataset with unlabelled data. Our method alleviates unique challenges histopathological imaging practice: large-scale information, costly manual anno- tation, and protective data handling. The biological plausibility of DiftInfinite data is evaluated in survey by ten experienced pathologists well downstream classification and segmentation task. Samples from the model score strongly on anti-copying metries which is relevant for the protection of patient data.
a)
Synthetic
Real
b)
Synthetic
Real
Alveole
Artery
Artifacts
Carcinoma
Cartilage
Connective Tissue
Necrosis
Tumor Stroma
Figure 1: a) Examples of synthetic and real 2048 x 2048 images. b) Pairs of 512 x 512 synthetic tiles (top) with the closest real images found with Inception-v3 near-neighbour (bottom).
Introduction
Deep learning (DL) models are promising auxiliary tools for medical diagnosis [1-3]. Applications like segmentation and classification have been refined and pushed to the limit on natural images [4]. However, these models trained on rich datasets still have limited applications medical data. While segmentation models rely on sharp object contours when applied to natural data, in medical imaging, the model struggles detect specific feature because has a "limited ability handle objects with missed boundaries" and often "miss tiny and low-contrast objects" [5, 6]. Therefore, task-specife medical applications require their own specialised and fine-grained annotation. Data labelling is arguably one of the most critical bottlenecks in healthcare machine learning (ML) applications. In histopathology, pathologists examine the histological slide at multiple levels, usually starting with lower magnification to analyse the tissue architecture and cellular arrangement and gradually proceeding to a higher magnification to examine cell morphology and subcellular features, such as the appearance and number of nuclcoli, chromatin density and cytoplasm appearance. Annotating features within gigapixel whole slide images (WSIs) with this level of detail demands effort and time, often leading to sparse, limited annotated data. In addition, due to privacy regulations and ethics [7, 8), having access to medical data can be challenging since has been shown that it is possible to extract patients' sensitive information [9] from this data.
In histopathology, ttateoof-the-an ML models require the context the entire WSIs, with features at different scales, in order to distinguish between different tumor sub-types, grades and stages [10]. Despite the demonstrated effectiveness of diffusion models (DMs) in generating natural im- ages compared to other approaches, they still have rarely been applied in medical imaging. Existing

---

generative models in histopathology can generate images of relatively small resolution compared to WSIs. To give a few examples, the application of Generative Adversarial Networks (GANs) in cervical dysplasia detection [11], glioma classification [12], and generating images of breast and colorectal cancer [13], generate images with 256 x 128 px, 384 384 px and 224 x 224 px, respec- tively. In spite of their current limitations in generating images at scales necessary to fully address all medical concerns, the use of synthetic data in medical imaging can provide a valuable solution to the persistent issue of data scarcity [14-17]. Models generally improve after data augmentation and synthetic images are equally informative as real images when added to the training set [18, 19]. Data augmentation could also help with the underrepresentation in data sets of rare cancer subtypes. By adding synthetic images to the training set, Chen et al. [20] demonstrated that their model had better accuracy in detecting chromophobe renal cell carcinoma, which is a rare subtype of renal cell carcinoma. Furthermore, Doleful et al. [21] showed how synthetic histological images could be used for educational purposes for pathology residents. Regarding the challenges highlighted before, we present a novel sampling method to generate large histological images with long-range pixel correlation (see Fig. 1), aiming extend up to the resolution of the WSI.
Our contributions are as follows: 1) We introduce DiffInfinite, a hierarchical generative framework that generates arbitrarily large images, paired with their segmentation masks. 2) We introduce fast outpainting method that can be efficiently parallelized. 3) The quality of DiffInfinite data evaluated by ten experienced pathologists as well as downstream machine learnings tasks (classification and segmentation) and anti mnti-duliccaion metrics to assess the leakage of patient data from the training set.
2 Related Work
Large-content image generation can be reduced to inpainting/outpainting tasks. Image inpainting is the problem of reconstructing unknown or unwanted areas within an image. A closely related task is image outpainting, which aims to predict visual content beyond the boundaries of an image. In both cases, the newly the rest of the image. Such image completion approaches can help utilise models trained on smaller patches for the purpose of generating large images, by initially generating the first patch, followed by its extension outward in the desired direction.
outpainted image regions have to be visually indistinguishable with respect to
Traditional approaches Traditional methods for image region completion rely on repurposing known image features, necessitating costly nearest neighbour searches for suitable pixels patches [22-26]. Such methods often falter with complex or large regions [24]. In contrast, DL enables novel, realistic image synthesis for inpainting and outpainting. Some methods like Deep Image Prior [27] condition new image areas on the existing image, while others aim to learn natural image priors for realistic generation [28, 29].
Generative modelling for conditional image synthesis GANs have dominated image-to-image translation tasks like inpainting and outpainting for years [28-42]. Recently, DMs have surpassed GANs in various image generation tasks [43]. Palette [44] was the first to apply DMs to tasks like inpainting and outpainting. RePaint [45] and ControlNet [46] demonstrate resampling and masking techniques for conditioning using a pre-trained diffusion model. SinDiffusion [47] and DiffCollage [48] offer oftee-of-the-ar outpainting solutions using DMs trained with overlapping patches. In parallel to our work, Bond-Taylor and Willcocks [49] developed related appreach called oD-Diff which trains on random coordinates, allowing the generation of sampling. However, contrast to our approach the method does not involve image compression in a latent space.
images during
Synthetic data assessment The authenticity of synthetic data produced by DMs, trained on vast paired labelled datasets [50], remains contentious. Ethical implications necessitate distinguishing generated images are replicas of training data [51, 52]. The task is complicated due to subjective vi- sual similarities and diverse dataset ambiguities. Various metrics have been proposed for quantifying data replication, including information theory distances from real data [53], consistency measure- ments using downstream models [54, 55), comparison with inpainted areas [52], and detection of "forgotten"" examples [56].
3

---

Next step
E+(-)
EØ
24
++ X
Timestep menkong
Randomly pick region of interest
Diffusion Submask CN
E
(c)
Timestess Piael
Diffusion steps
Whole image
Random Patch-T)
Figure 2: Difflnfinite generation method. a) Large-scale context mask generation. diffusion model conditioned on large-scale conditional prompt (e.g. Adenocarcinoma subtype) generates low-resolution mask. The mask is upsampled via linear interpolation to the desired image size. b) Diffusion steps on large images. Given random position, we select sub-tile with its segmentation mask. A diffusion model generates in parallel the next step conditioned on each conditional label, or prompt, found in the mask. The outputs are masked individually with the corresponding label. The next step is the union of all the sub- batchat c) Tracking time steps pixel-wise We keep track of the time step of each pixel in the large image. The model evolves only the pixels with the higher time step on each iteration.
3 Preliminaries
Diffusion Models DMs [57-59] represent class of parameterized Markov chains that effectively optimize the lower variational bound associated with the likelihood function of the unknown data distribution. By iteratively adding small amounts of noise until the image signal is destroyed and then learning to reverse this process, DMs can approximate complex distributions much more faith- fully than GANs [60]. The increased diversity of samples while preserving sample fidelity comes at the cost of training and sampling speed, with DMs being much slower than GANs [43]. The universally adopted solution to this problem is to encode the images from pixel space into lower dimensional latent space via a Vector Quantised- Variational AutoEncoder (VQ-VAE), and perform the diffusion process Denoising Diffusion Implicit Models (DDIMs) sampling method [62] leads to faster sampling while preserving the DM objective
the latents, before decoding back pixel space [61]. Pairing this with the
21
VI (1C0(22,1) + v a - -f dees Va
21-1 = 0
(1)
where 2t is the latent variable at time step in the VQ- latent space, ar is the noise scheduler. is the noise learned by the model and e; is random noise. Conditioning can be achieved either by specifically feeding the condition with the noised data [44, 63], by guiding an unconditional model using external classifier [64, 65] by fassifier-frre guidance [66] used in this work, where the

---

convex combination
() (1+ w)) we)(2) Ø).
(2)
of a conditional diffusion model co(2e, c) and an unconditional model co(2g, 2)) is used for noise
estimation. The parameter w controls the tradeoff between conditioning and diversity, since w > introduces more diversity in the generated data by considering the unconditional model while 0
uses only the conditional model.
Infinite Diffusion
The DiffInfinite approach we present here!, is generative algorithm to generate arbitrarily large images without imposing conditional independence, allowing for long-range correlation structural information. The method overcomes this limitation of DMs for large-content generation by deploy- ing multiple realizations of a DM on smaller patches. In this section, we first define mathematical description of this hierarchical generation model and then describe the sampling method paired with masked conditioned generation process.
4.1 The Method
Let X X' be large-content generating random variable taking values in RDD Using the ap- proach of latent diffusion models [61]. the high-dimensional content is first mapped to the latent space TR. D by ((X) = Y ~ Yo. For simplicity, an ideal encoder-decoder pair (Φ, v) such that v (X)) = X is the identity on RKD Assume further, to have reverse time model (SM Ea) at hand consisting of sampling method S.M and learned model trained small patches Z Z taking values in Rd The reverse time model transforms 2T ~ NN(0 Ia) over the time steps t E (T,T - I.., 1} recursively by
assume throughout this work the existence of
-t-1
SMo(2)
to an approximate instance of Z4. We aim to sample instances from V' by deploying multiple realizations of the reverse time model (SMo, Ea). Towards that goal, define the of projections
:= {projr: RDD _ I N correspond to d indices of connected pixels in RP), (4)
where proj e C models crop pY) R& of connected pixels from the latent image Y. Since the model is trained on images taking values in Rd the standing assumption is
Assumption Any projection proj E C maps Y to the same distribution proj(Y) ~ Zo in Rd
Since the goal is to approximate an instance of Ve, we initialize the sampling method by yr N(O, ID) and proceed in the following way: Given yt, randomly choose proji,,..., rrrjjj... e C independent of the state Yr such that projm,...,projj proji.. are non equal crops that cover all latent pixels in RDD. To be more precise, for every i (1,..., D3 we find at least one j E ..., {I...,, with i Jj- For every projection projn. .rrojjt,, we calculate the crop z! projr, (ye) of the current state and perform one step of the reverse time model following the sampling scheme
+ SMo(z()
(1...,
This results in overlapping estimates 21 2 21=h, of the subsequent state t-1 and we simply assign to every pixel in the latent space the first value computed for this pixel such that
[ 1
1
where j = min { Lie |
(6)
and refers to the entry in 2/_1 corresponding i with [projı, (ye-1)}e [ve-1] 1]: Hence, starting from yr AC ID) sample in the first step from distribution
yr-1 ~ P1- yr, proji.
(7)
Using Bayes' theorem, this distribution simplifies
P yr,
Pr-1.e
(8)
'Code available https://githhb. http://hhhhhh.oooooooco/ooo/aaat
5

---

since we sample the projections independently from yr. Repeating the argument, we sample in every step from a distribution 9t-1 Pt-1,o yr) over Rn instead of sampling from 2t-1 1 (2)2220((2222_. 2,,, 2r) over RR. Hence, we approximate the true latent distribution V by the approximate distribution with density po.o(vlvi, .... Vr). In contrast to [48], our method does not use the assumption of conditional independence and the method can be applied to wide range of DMs, without an adjustment of the training method. As the authors of [48] point out in their section on limitations, the assumption of conditional independence not well-suited in cases of a data distribution with long-range dependence. For image generation in the medical context, we aim to circumvent this assumption as we not want to claim that the density of given region depends only neighboring region. The drawback of dropping the assumption is that we only approx- imate the reverse time model of the latent image distribution Ye indirectly, by multiple realizations of reverse time model that approximates Zo.
4.2 Semi-supervised Guidance
In order to generate diverse high-fidelity data, DMs require lots of training data. Perhaps, train- ing few samples still extracts significant features but it lacks variability, resulting in simple replicas. Here, we show how to enhance synthetic data diversity using classifier-fi free guidance as a semi-supervised learning method. In the classifier-free free guidance [66]. single model is trained con- ditionally and unconditionally on the same dataset. We adapt the training scheme using two separate datasets. The model is guided by small and sparsely annotated dataset Q1, used for the conditional training step, while extracts features by the large unlabelled dataset go. used on the unconditional training step (see 1lg.1)
[E0. 2) (0(2))
if 3 Punc (20. c) ~ Q1 (z0; c) otherwise
(zo, c)
where u is sampled from a uniform distribution in [0,1], Punc is the probability of switching from the conditional to the unconditional setting and is null label. During the sampling, tradeoff between conditioning and diversity is controlled via the parameter in eq.2.
4.3 Sampling
High-level content generation The outputs of DMs have pixel consistency within the training image size. Outpainting an area with a generative model might lead to unrealistic and odd artifacts due to poor long-range spatial correlations. Here, we show how predict pixels beyond the image's boundaries by generating a hierarchical mapping of the data. The starting point is the generation of the highest-level representation of the data. In our case, the sketch of the cellular arrangement details are unnecessary at this stage, we can
in the WSI (see Figure 2a). Since downsample the masks until the clustering pattern still recognizable. The diffusion model, condi- tioned on the context prompt (e.g. Adenocarcinoma subtype). learns the segmentation masks which contain the cellular macro- strucurrs information.
Random patch diffusion Given a segmenta- tion mask M, we can proceed with the large image sampling according to Section 4.1 in the latent space TDD of Y Alg.2). Since trained conditional dif- fusion model with conditions Ci,..., CN. the learned model takes the form c0(Ze,t) (co tlcı),..., co(2s, t(cx)). Given yts we first sample projections proji, ..., proji.. C, corre- sponding to different crops d connected pix- els up to the m-th projection with ofht
DiCcallgee
ofhifiibeio (Durs)
10a
(xX) (see
55000
Comparison of sampling speed for DiffCollage and Difflnfinite, measuring diffusion steps required for image sampling. Demonstrating increased efficiency of Difflnfi- nite for larger images.
Figure 3:
(I,..., D} and T "د U I, + Ø (see the left hand-side of Figure 2b). Note that m is not fixed, but varies over the sampling steps and is upper bounded by the number of possible crops of d connected pixels. The random selection of

---

the projection is implemented such that regions with latent pixels of low projection coverage are more likely. Secondly, we calculate for every projection j {l....,m) m] the crop projr, (st) and perform one step of the DDIM sampling procedure using the classifier-free guidance model (1 + ()) c) wee f, Ø), where Eq is the learned model and 2ं projr, (2t). This results for every pixels i E I in N values DDIM c.i (projr, (3t))... ..., DDIMo oN (projz, (v0)), one for every condition 6; (see the right hand-side of Figure 2b). If $ T for all j'. the pixel i has not been considered yet and we assign the value [yr-1] to the pixel i under the projection 1j and M, is the value of in the mask M. Since we are updating random projections of the overall image, in the t-th step pixels either have the time index t or + 1, resulting in a reversed diffusion process of differing time states. We initialize a tensor L,, with the same size D as the latent variable, to keep track of the time index for each pixel. Each element is set to L1 = T. In the j-th iteration of the t-th step we only update the pixels that have not been considered in one of the previous iterations of the t-th diffusion step, hence all the pixels in i 13 with (rrojr,(Tx))
[DDT MM,M, Mr (projr,(V)) (V,))], where / corresponds
+ 1, similarly to the inpainting mask in the Repaint sampling method [45].
To restore the pixels that already received an update, i.e. every pixel i E I with projr, (Lr); t. we store a replica of the pre- vious diffusson step for every pixel. Finally, we update all the time states in Le that received an update in the j-th iteration to t resulting in proji, (Li); = for all Iz. See the top row of Fig.2c for an illustration of the evolu-
Algorithm DiffInfinite Training
Repeat
1: Randomly train lahelled or unlabelled data with probubiiity Punce
~ fniifrm [0. 1]
(sp. c) [=0, 91 (#D: e)
(Ro, 2) ~ (o()))
if u 2 Pane atherwise
2: Sample midds time step
t- Uniform (L..... T)
3: Sumple noise, ~ (O00
tion of T.. The random patch diffusion can also be applied to mask generation, where the only condition is the context prompt. This method can generate segmentation masks of arbitrary sizes with the correlation length bounded by
4: Comups datt, 7e300 .ree
5: Take gradient descent stept
Valle cg(se,
6: uutil couverged
Algorithm 2 DiffInfinite Sampling
two times the training mask image size.
Input: Highi-loell segmentation mask A a RP and leamed model eg Output: Syathetic image X with the mask size Initialization:
Parallelization The sampling method pro- posed has several advantages. In Zhang et al. [48] each sequential patch is outpainted from
Vr AO( 1),, index set Jn Ø and state tensor Ly =
Repeat
I:
for (T 1,...0) do
the previous one with 50% of the pixels shared. Here, the randomization eventually leads to every possible overlap with the neighboring patches. This introduces longer pixel correla- tion across the whole generated image, avoid-
2:
while uo + (1..... D) do m +1 Select candonll prosim C' projim
5:
Crop =?"
profimm (81)
6:
all conditions (1,....
7:
DDIM campling with classilen-fee gridince
ing artifacts due to tiling. In Figure 3, we
8:
=?" en "ro."=l en)
show that the number of steps in the whole 9:
end for
large image generation process is drastically re- duced with the random patching method with 12:
10:
for indices e Im do
11:
it I2 forall < m then
[st-t] - [=]"]]ād,]] projr.. (I.)). - A
respect to the sliding window one. Morcover, 13:
in the sliding sampling method, the model can
14:
such that [pragr,,, (se-1)le Dll [M
be paralleled only or 4 times, depending if
15:
end
are outpainting the image horizontally or both axis. In our approach, we can parallelize the sampling up to the computational resource
end for
end while
18:
end for
19:
limit.
Data Assessment
To assess synthetic images for medical image analysis, we need to take various dimensions of data assessment into account. We extend traditional metrics from the natural image community with qualitative and quantitative assessments specific to the medical context. For the qualitative analysis, a team of pathologists evaluated the images for histological plausibility. The quantitative assessment entailed a proof-of-concept that model can learn sensible features from the synthetically generated image patches for relevant downstream task, As data protection is highly relevant regarding patient data, we performed evaluations to rule out memorization effects of the generative model.
7

---

Table 1: Metries to quantitatively evaluate the quality of the generated images. Left: scores for images of size 512 x 512. Difflnfinite (a) first generates a mask and secondly an image following Section 4.1. Right: scores for real and generated images of size 2048 x 2048 resized 512x All methods use the same model trained on small patches of size 512 x 512. DiffCollage corresponds to the method proposed in [48]. Diftlnfinite (b) uses the real masks, while DiffInfinite (c) first generates mask and secondly the large image. DiftInfinite (b) & (c) refers to the mixture of the generated
dataset from DiffInfinite (b) and DiffInfinite (c).
IP f
IR
IS 1
FID
IP
IR 1
Morph-Diffusion (72)
0.26
0.85
2.1
20.1
DiffCollage
0.94
0.22
DiffInfinite (b)
0.95
0.48
NASDM (73]
2.7
15.7
DiffInfinite (c)
0.98
0.44
Difflnfinite (a)
0.94
0.70
2.7
26.7
DiflInfinite (b) (c)
0.98
0.33
5.1 Traditional Fidelity
We evaluate the fidelity of synthetic 512 x 512 images by calculating Improved Precision (IP) and Improved Recall (IR) metrics between 10240 real and synthetic images [67].² The IP evaluates syn- thetic data quality, while the IR measures data coverage. Despite their unsuitability for histological data [68, 691. Srechet-Inception Distance (FID) and Inception Score (IS) [70, 71] are reported for comparison with [72] and Shrivastava and Fletcher [73].' The metrics' explanations and formulas can be found in Appendix C.
In Table 1 (left), we report an IP of 0.94 and an IR of 0.70, indicating good quality and coverage of the generated samples. However, we note that these metrics are only somewhat comparable due to the different types of images generated by MorphDiffusion [72] and NASDM [73]. For the large images of size 2048 2048, we rely solely on the IP and IR for quantitative evaluation due to the limited number of 200 generated large images. As shown in Figure 3(a) of [67], FID is unsuitable for evaluating such a small sample size, while IP and IR are more reliable. In Table (right), we find that generating images first results in slightly higher IR, while generating the mask first achieves an IP of 0.98. For the sake of completeness we also report the scores then combining the two datasets. To compare our method to DiffCollage better than DifiCollage wrt. to IP and IR. The drop of IR to 0.22 might be a result of the tiling artifacts observable in the LHS of Figure 11.
generate 200 images using [48]. DiffInfinite performs
5.2 Domain Experts Assessment
To assess the histological plausibility of our generated images, we conducted a survey with co- hort of ten experienced pathologists, averaging 8.7 years of professional tenure. The pathologists were tasked with differentiating between our synthetized images and real image patches extracted from whole slide images. We included both small patches (512 x 512 px) as commonly used for downstream tasks as well as large patches (2048 2048 px). Including large patches enabled us to additionally evaluate the modelled long-range correlations terms of transitions between tissue types as well as growth patterns which are usually not observable on the smaller patch sizes but total the survey contained 60 images, in equal parts synthetic and real images as well as small and large patches. The overall ability of pathologists to discern between real and synthetic images was modest, with an accuracy of 63%, and an average reported confi- dence level of 2.22 on 1-7 Likert scale. While we observed high inter-rater variance, there was no clear correlation between experience and accuracy (1(8) = .069, p5585), nor between confidence level and accuracy (r(8)= = ,446, 197). Furthermore there was no significant correlation between the participants' completion time of the survey and the number of correct responses (r(8) -08, p=.826).
essential histopathology.
Surprisingly, we found a similar performance for both, real and synthetic images. This indicates that, while clinical practice is mostly based on visual assessment, it is not a common task for pathologists to be restricted to parts of the whole slide image only. More detailed visualizations of the individual
scores can be found in Appendix B. Besides this satisfactory result, we additionally wanted to
and--nnnndddrralll-m www.eeed-treleeeerrrrrrtttrr -zetric-pytorch
8

---

Table 2: Zero-shot evaluation results of the downstream tasks, encompassing both classification and segmentation scenarios. We employed three distinct models for each scenario: The first, "Trained Real," was trained using real data (in-house IH1), which also served as the training set for DiffInfi- nite. The second, "Trained Synthetic," was trained using samples generated from Difflnfinite, and the third, "Trained Augmented," utilized a combination of real and synthetic data. Our evaluation extends across separate lung cohorts (internal datasets IH2 and IH3) and additional indications (ex- ternal datasets NCT, CRC, PCam), with varying degrees of data drift introduced.
IHI
IH2
IH3
NCT-100K
CRC-7K
PCam-327K
Patient chunge Different center
Patient change
Patient change
Different center
Different center
Drift components
Indication change
Indication change
Lower resolution
Trained Real
0.846 + 0.005
0.733 + 0.021
0.598 + 0.049
0.857 + 0.009
0.822 + 0.034
0.628 + 0.035
Trained Synthetic
0.717 + 0.025
0.753 + 0.005
0.699
0.002
0.796 0.023
0.753 0.038
0.628 0.012
Trained Augmented
0.852 0.007
0.732 0.027
0.637 1 0.025
0.847 0.044
0.811 0.057
0.641 1 0.035
(a) Classification results
IH2
Trained Real
0.614 +: 0.009
Trained Synthetic
0.471 +: 0.039
Trained Augmented
0.710 + 0.021
(b) Segmentation results
explore the limitations of our method by assessing the nuanced differences pathologists observed between synthetic and real images. While overall the structure and features seemed similar and hard to discern, they sometimes reported regions of inconsistent patterns, overly homogencous chromatin in some of the synthetic nuclei, peculiarities in cellular and intercellular structures, and aesthetic el- ements. These seemed to be especially pronounced in tumorous regions where sometimes the tissue architecture appeared exaggerated, the transition to stroma surrounding tissue was too abrupt and some cells lacked distinguishable nucleoli or cytoplasm. We attribute the nuanced effect of larger image size the accuracy this observation (ef. Fig. 5C). Overall the finding of the conducted survey demonstrates how complex the task of distinguishing between real and synthetically gener- ated data is even for experienced pathologists while still highlighting potential areas to improve the generative model.
5.3 Synthetic Data for Downstream Tasks
A major interest in the availability of high quality labeled synthetic images their use in downstream digital pathology applications. In this area, two primary challenges are the binary classification of images into cancerous or healthy tissues and the segmentation of distinct tissue areas the tumor microenvironment. The unique ability of our technique to generate images of different cancer sub- types through the context prompt as well as the ability to create new segmentation masks and their corresponding H&E images specifically addresses these two challenges. Notably, expert annota- tions are costly and time consuming to acquire thus emphasizing the benefits of being able to train on purely synthetic datasets or augmenting annotated data in the low data regime. To showcase these two usecases we performed a series of experiments in both classification and segmentation settings. For all experiments, we trained baseline classier on a relatively small number of expert annotations IHI (#patches = 3726)) - the same that were used to train DiffInfinite - and additionally trained one model purely on synthetic data (IH1-S, #patches 9974, w 0), and one model the real data augmented with the synthetic images. To generate target labels for the classification experiments, we simplified the segmentation challenge by categorizing patches with at least 0.05% of pixels labeled as "Carcinoma" in the segmentation masks as 'Carcinoma'. All other patches were labeled 'Non- Carcinoma'. We evaluated all three classification models on several oul-of-distribution datasets. We utilized two proprietary datasets (from the same cancer type with similar attributes but from distinct patient groups: IHI (# patients=13, patches=704) and IH3 (# patients=2, # patches=2817), More- over, we assessed the models using two public datasets (NCK-CR) [74] and PatchCamelyon [75]), both representing tissue from different organs with distinct morphologies. Our findings, summarized in Table 2a, suggest that classifier's distr--disttibttion performance, trained with limited sample size and morphological diversity, can vary significantly (ranging from 0.628 to 0.857 balanced ac- curacy). This variability cannot be attributed solely to morphology but may also be influenced by

---

factors such as resolution and variations in scanning and staining techniques. Training exclusively with a larger set of synthetic images can enhance performance on some datasets (specifically IH2 and IH3), underscoring the advantages of leveraging the full training data in semi-supervised man- ner within the generative model. Incorporating synthetic data as an augmentation to real data not only prevents the classifier's performance decline, as seen on NCT-CRC and Patchcamelyon on similar datasets but also bolsters its efficiency on more distinct ones. For the more challenging segmentation task we again trained three segmentation models to differentiate between carcinoma, stroma, necrosis, and a miscellaneous class that included all other tissue types, such as artifacts. The baseline performance of the real data model Fi score of 0.614 0.009 (across three random seeds) highlights the difficulty of generalizing out of distribution in this tasks. While the purely synthetic model was not able to fully recover the baseline performance (0.471 + 0.039), augmenting the small annotated dataset with synthetic data enhanced predictive performance an Fi score of 0.710 + 0.021. This boost of 10 percentage points in per- formance demonstrates that the synthetic data provide new, relevant information to the downstream task. In summary, our findings demonstrate the feasibility of meeting surpassing baseline per- formance levels for both tasks using either entirely synthetic data or within an augmented context. Nevertheless, the advantages of employing synthetic data in downstream tasks continue to pose a challenge, not only within the medical image domain but also across various other domains [76-78), thus requiring more comprehensive assessment and thorough examination.
distinct group of lung patients (dataset IH2) of
5.4 Considerations on Memorization
In medicine the adherence to privacy regulations is a sensitive requirement. While it is generally not possible for domain experts to infer patient identities from the image content of histological tile or slide alone [79], developers and users of generative models are well advised to understand the risk of correspondence between the training data and the synthesized data. To this end, we evaluate the training and synthesized data against two memorization measures. The authenticity score A E [0, 1] by [54] aims to measure the rate by which a model generates new samples (higher score means more innovative samples). Similarly, [80] aims to estimate the degree of data copying Or from the training data by the generative model. A CT < implies data copying, while Cr 0 implies an underfitting of the model. The closer to O the better. See Appendix C for precise closed form of the measures and Table for the full quantitative results, indicating that the DiffInfinite model is not prone to data copying across all resolutions and variations considered here 4. The A range between 0.86 and 0.98, signifying a high rate of authenticity. While other papers unfortunately do not report such detailed memorization statistics for their models, the results by [54] suggest that score > 0.8 is not trivial to achieve. None of the models under consideration [54] (VAE, DCGAN, WGAN-GP, ADS- ADS-GAN) achieve more than 0.82 in A on simpler data (MNIST). This interpretation is strengthened by the results of Cr >> 0 which indicates that the model might even be underfitting and is not in a data copying regime. Qualitative results the nearest neighbour search between training and synthetic data in Figure further corroborate these quantitative results.
6 Conclusions
DiffInfinite offers a novel sampling method to generate large images in digital pathology. Due to the high-level mask generation followed by the low-level image generation, synthetic images contain long-range correlations while maintaining high-quality details. Since the model trains and samples on small patches, it can be efficiently parallelized. We demonstrated that the classifier-fres guidance can be extended to a semi-supervised learning method, expanding the labelled data feature space with unlabelled data. The biological plausibility of the synthetic images was assessed in survey by 10 domain experts. Despite their training, most participants found it challenging to differentiate between real and synthetic data, reporting an average low confidence in their decisions. We found that samples from DiffInfinite can help in certain downstream machine learning tasks, on both in- as well as -ut-o--distribution datasets. Finally, authenticity metrics validate DiffInfinite's 's capacity to generate novel data points with little similarity to the training data which is beneficial for the privacy preserving use of generative models in medicine.
'We use https: http:///tttuubccoooo//ooooooooo/ from [81] to calculate both scores.
10

---

7 Acknowledgements
We would like to acknowledge our team of pathologists who provided valuable feedback in and outside of the conducted survey special thank you to Frank Dubois, Niklas Prenissl, Cleopatra Schreiber, Vitaly Garg, Alexander Arnold, Sonia Villegas, Rosemarie Krupar and Simon Schallen- berg. Furthermore, we would like thank Marvin Sextro for his support in the analyses. This work was supported by the Federal Ministry of Education and Research (BMBF) as grants [SyReal (011821069B) RM-S is grateful for EPSRC support through grants EP/T00097X/1, EP/R01 8634/1 and EP/T021020/1, and DI for EP/R513222/1. MA is funded by Dotphoton, QuantIC and UofG Ph.D. scholarship.
References
[1] Ravi Aggarwal, Viknesh Sounderajah, Guy Martin, Daniel SW Ting, Alan Karthikesalingam, Dominic King, Hutan Ashrafian, and Ara Darzi. Diagnostic accuracy of deep learning in medical imaging: systematic review and meta-analysis. NPJ digital medicine, 4(1):65, 2021.
[2] Antonio Parziale, Monica Agrawal, Shengpu Tang, Kristen Severson, Luis Oala, Adarsh Sub- baswamy, Sayantan Kumar, Elora Schoerverth, Stefan Hegselmann. Helen Zhou, Ghada Za- mzmi, Purity Mugambi, Elena Sizikova, Girmaw Abebe Tadesse, Yuyin Zhou, Taylor Killian, Haoran Zhang, Fahad Kamran, Andrea Hobby, Mars Huang, Ahmed Alaa, Harvineet Singh, Irene Y. Chen, and Shalmali Joshi. Machine learning for health (ml4h) 2022. In Antonio Parziale, Monica Agrawal, Shalmali Joshi, Irene Y. Chen, Shengpu Tang, Luis Oala, and Adarsh Subbaswamy, editors, Proceedings of the 2nd Machine Learning for Health sympo- sium, volume 193 of Proceedings of Machine Learning Research, pages 1-11. PMLR, 28 Nov 2022.
[3] Maximilian Springenberg, Annika Frommholz, Markus Wenzel, Eva Weicken, Jackie Ma, and Nils Strodthoff. From modern cnns to vision transformers: Assessing the performance, robust- ness, and classification strategies of deep learning models in histopathology. Medical Image Analysis, 87:102809, 2023, ISSN 1361-8415.
[4] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint wrXiv:2304.03284 2023.
[5] Miriam Hägele, Philipp Seegerer, Sebastian Lapuschkin, Michael Bockmayr, Wojciech Samek, Frederick Klauschen, Claus-Robert Müller, and Alexander Binder. Resolving challenges in deep learning-basee analyses of histopathological images using explanation methods. Scien- tific reports, 10(1):1-12, 2020.
[6] Jun Ma and Bo Wang. Segment anything in medical images. arXiv preprint arXi::330.113066, 2023.
[7] Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al. The future of digital health with federated learning. NPJ digital medicine, 3(1):119 19, 2020.
[8] Luis Oala, Jana Fehr, Luca Gilli, Pradeep Balachandran, Alixandro Werneck Leite, Saul Calderon- Ramirezz Danny Xie Li, Gabriel Nobis, Erick Alejandro Muñoz Alvarado, Gio- vanna Haamilo-GGuterree. Christian Matck, Arun Shroff, Ferath Kherif, Bruno Sanguinetti, and Thomas Wiegand. MI4h auditing: From paper to practice. In Emily Alsentzer, Matthew B. A. McDermott, Fabian Falck, Suproteem K. Sarkar, Subhrajit Roy, and Stephanie L. Hy- land, editors, Proceedings of the Machine Learning for Health NeurIPS Workshop, volume 136 Proceedings of Machine Learning Research, pages 280-317, PMLR, 11 Dec 2020.
[9] Christopher G Schwarz, Walter K Kremers, Terry M Therneau, Richard R Sharp, Jeffrey Gunter, Prashanthi Vemuri, Arvin Arani, Anthony J Spychalla, Kejal Kantarci, David S Knop- man, et al. Identification of anonymous mri research participants with face-recognition soft- ware. New England Journal of Medicine, 381((7))168441688 2019.
11

---

[10] Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchi- cal self-supervised learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16144-16155, 2022.
[11] Yuan Xue, Jiarong Ye, Qianying Zhou, Rodney Long, Sameer Antani, Zhiyun Xue, Carl Cornwell, Richard Zaino, Keith C Cheng, and Xiaolei Huang. Selective synthetic augmenta- tion with histogan for improved histopathology image classification. Medical image analysis, 67:101816, 2021.
[12] Le Hou, Ayush Agarwal, Dimitris Samaras, Tahsin M Kurc, Rajarsi R Gupta, and Joel H Saltz. Unsupervised histopathology image synthesis. arXiv preprint arXiv:1112.05021, 2017.
[13] Adalberto Claudio Quiros, Roderick Murray-Smith, and Ke Yuan. PathologyGAN; learning deep representations of cancer tissue. Journal of Machine Learning for Biomedical Imaging, 4:1-48, 2021.
[14] Ahmad B Qasim, Ivan Ezhov, Suprosanna Shit, Oliver Schoppe, Johannes Johannee Paetzold, Anjany Sekuboyina, Florian Kofler, Jana Lipkova, Hongwei Li, and Bjoern Menze. Red-gan: Attack- ing class imbalance via conditioned generation. yet another medical imaging perspective. In Medical Imaging with Deep Learning, pages 655-668. PMLR, 2020.
[15] Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Matgorzata Potacin, Juan Manuel Zambrano Chaves Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P Langlotz, and Akshay Chaudhari. Roentgen: Vision-language foundation model for chest x-ray generation. arXiv preprint arXiv:2211. 2022.
[16] Pierre Chambon, Christian Bluethgen, Curtis P Langlotz, and Akshay Chaudhari. Adapting pretrained vision-language foundational models medical imaging domains. arXiv preprint arXiv:22 10.04133. 2022.
[17] Luis Oala, Marco Aversa, Gabriel Nobis, Kurt Willis, Yoan Neuenschwander, Michèle Buck, Christian Matek, Jerome Extermann, Enrico Pomarico, Wojeiech Samek, Roderick Murray- Smith, Christoph Clausen, and Bruno Sanguinetti. Data models for dataset drift controls in machine learning with optical images. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
[18) Adrian B Levine, Jason Peng. David Farelll, Mitchell Nursey, Yiping Wang, Julia R Naso, Hezhen Ren, Hossein Farahani, Colin Chen, Derek Chiu, Aline Talhouk, Brandon Sheffield, Maziar Riazy, Philip P Ip. Carlos Parra-Herran, Anne Mills, Naveena Singh, Basile Tessier- Clouticr, Taylor Salisbury, Jonathan Lee, Tim Salcudean, Steven JM Jones, David Hunts- man, C Blake Gilks, Stephen Yip, and Ali Bashashati. Synthesis of diagnostic quality cancer pathology images by generative adversarial networks. The Journal of Pathology, 252( 188, 2020.
[19] Virginia Fernandez, Walter Hugo Lopez Pinaya, Pedro Borges, Petru-Daniel Tudosiu, Mark Graham, Tom Vercauteren, and M Jorge Cardoso. Can segmentation models be trained with fully synthetically generated data? In Simulation and Synthesis in Medical Imaging: 7th International Workshop, SASHIMI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings, pages 79-90. Springer, 2022.
[20] Richard Chen, Ming Lu, Tiffany Chen, Drew Williamson, and Faisal Mahmood. Synthetic data in machine learning for medicine and healthcare, Nature Biomedical Engineering, 5:1-5, 06 2021.
[21] James M Dolezal, Rachelle Wolk, Hanna M Hieromnimon. Frederick M Howard, Andrew Srisuwananukorn, Dmitry Karpeyev, Siddhi Ramesh, Sara Kochanny, Jung Woo Kwon, Meghana Agni, et al. Deep learning generates synthetic cancer histology for explainability and education. NPJ Precision Oncology, 7(1):49, 2023.
[22] Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1033-1038. IEEE, 1999.
12

---

[23] Li-Yi Wei and Mare Levoy. Fast texture synthesis using tree- structurdd vector quantization. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 479-488, 2000.
[24] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image in- painting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 417-424, 2000.
[25] Zongben Xu and Jian Sun. Image inpainting by patch propagation using patch sparsity. IEEE transactions on image processing, 1955):115311165, 2010.
[26] Lin Liang, Ce Liu, Ying-Qing Xu, Baining Guo, and Heung- Yeung Shum. Real-time texture synthesis by patch-based sampling. ACM Transactions on Graphics (ToG), 2073):27-1500, 2001.
[27] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446-9454 2018.
[28] Avisek Lahiri, Arnav Kumar Jain, Sanskar Agrawal, Pabitra Mitra, and Prabir Kumar Biswas. Prior guided gan based semantic inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13795-13705, 2020.
[29] Liang Liao, Jing Xiao, Zheng Wang, Chia- Lin, and Shin' ichi Satoh. Guidance and eval- uation: Semantic-aware image inpainting for mixed scenes. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pages 683-700. Springer, 2020.
[30] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Infor- mation Processing Systems, volume 27. Curran Associates, Inc., 2014.
[31] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Con- text encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536-2544, 2016.
[32] Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 5505-5514, 2018
[33] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. Proceedings of the IEEE/CVE international conference on computer vision, pages 4471 14880, 2019.
[34] Yi Wang, Ying-Cong Chen, Xin Tao, and Jiaya Jia. Venet: robust approach to blind im- age inpainting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt XXV 16, pages 752-768. Springer, 2020.
[35] Shengyu Zhao, Jonathan Cui, Yilun Sheng. Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2103. 10428, 2021.
[36] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lem- pitsky. Resolution-roburt large mask inpainting with fourier convolutions. arXiv preprint 27X1v:2109.07161, 2021.
[37] Mark Sabini and Gili Rusak. Painting outside the box: Image outpainting with gans. arXiv preprint arXiv: 08488, 2018
[38] Basile Van Hoorick. Image outpainting and harmonization using generative adversarial net- works, arXiv preprint arXiv: 1922119910009600 2019.
13

---

[39] Chich Hubert Lin, Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, and Ming- Ming-Hsuan Yang. InfinityGAN: Towards infinite-pixer pixell image synthesis. In International Conference on Learning Representations, 2022.
[40] Yen-Chi Cheng, Chieh Hubert Lin, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, and Ming- Hsuan Yang. Inout: diverse image outpainting via gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11431-11440, 2022.
[41] Yaxiong Wang, Yunchao Wei, Xueming Qian, Li Zhu, and Yi Yang. Sketch-guided scenery image outpainting, IEEE Transactions Image Processing, 30:2643-2222 2653,, 2021.
[42] Yaxiong Wang, Yunchao Wei, Xueming Qian, Li Zhu, and Yi Yang. Rego: Reference-guided outpainting for scenery image. arviv preprint arXiv:2 10601, 2021.
[43] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[44] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIG- GRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[45] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461- 11471, 2022.
[46] Lvmin Zhang and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[47] Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Sindiffusion: Learning diffusion model from single natural image. arXiv preprint 2rXiv:2211.12442 2022.
[48] Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming- Yu Liu. Diffcollage: Parallel generation of large content with diffusion models. arXiv preprint arXiv:2303. 17076, 2023.
[49] Sam Bond-Taylor and Chris G. Willcocks. oi-dfff: Infinite resolution diffusion with subsam- pled mollified states, 2023.
[50] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. urXiv preprint arXiv:22 10.00400 2022.
[51] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Dif- fusion art or digital forgery? investigating data replication diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2023.
[52] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito. and Eric Wallace. Extracting training data from dif- fusion models. arXiv preprint arXiv:2301. 13188, 2023.
[53] Nikhil Vyas, Sham Kakade, and Boaz Barak. Provable copyright protection for generative models. arXiv preprint axXiv:2302.10874 2023.
[54] Ahmed Alaa, Boris Van Breugel, Evgeny Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning, pages 290-306. PMLR, 2022.
[55] Luis Oala. Metrological machine learning (2MI)). 1 edition, 2023. URL https:// metrological.ng ml.
14

---

[56] Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. Measuring forgetting of memorized training examples. arXiv preprint arXiv:2207.00009, 2022.
[57] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynammis. In International Conference on Machine Learning, pages 2256-2265, PMLR, 2015.
[58] Yang Song and Stefano Ermon. Improved techniques for training score- generative mod- els. Advances in neural information processing systems,
2020.
[59] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In
H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851 Curran Associates, Inc., 2020.
[60] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[61] Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[62] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
[63] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-33, 2022.
[64] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc- Grew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 10741, 2021.
[65] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[66] Jonathan Ho and Tim Salimans. 2.X1::2007.1259. 2022.
Classifier-fre diffusion guidance. arXiv preprint
[67] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Buch E. Fox, and R. Garnett, editors, Advances in Neural Informa- lion Processing Systems, volume 32. Curran Associates, Inc., 2019.
[68] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes fréchet inception distance. In Proc. ICLR, 2023.
[69] Shane Barratt and Rishi Sharma. A note on the inception score. ArXiv, abs/1801.01973, 2018.
[70] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[71] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances Neural Information Processing Systems, vol- ume 29, Curran Associates, Inc., 2016.
[72] Puria Azadi Moghadam, Sanne Van Dalen, Karina C. Martin, Jochen Lennerz, Stephen Yip. Hossein Farahani, and Ali Bashashati. morphology focused diffusion probabilistic model for synthesis of histopathology images. arXiv preprint arXiv: arXiv: 200..222,.3377,, 2022.
15

---

[73]
Aman Shrivastava and P Thomas Fletcher. Nasdm: Nuclei-aware semantic histopathology image generation using diffusion models. arXiv preprint arXiv:2303.11477.222 2023.
[74] Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue, April 2018. URL http:://pooiii URL.htts::/doo..orr// wtt/oo...////1.11 zenodo. 1214456.
[75] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant enns for digital pathology. In Medical Image Computing and Computer Assisted Intervention-MICCA/ 2018: 21st International Conference, Granada, Spain, September 16- 20, 2018, Proceedings, Purt II 11, pages 210-218. Springer, 2018.
[76] Shenghuan Sun, Gregory M Goldgof, Atul Butte, and Ahmed M Alaa. Aligning synthetic med- ical images with clinical knowledge using human feedback. arXiv preprint arXiv:2306. 12438, 2023.
[77] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason We- ston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308. 06259, 2023.
[78] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training generated data makes models forget. arXiv preprint arXiv:2305. /793,, 2023.
[79] Petr Holub, Heimo Müller, Tomás Bíl, Luca Pireddu, Markus Plass, Fabian Prasser, Irene Schlünder, Kurt Zatloukal, Rudolf Nenutil, and Tomáš Brázdil. Privacy risks of whole-slide image sharing in digital pathology. Nature Communications, 14(1):2577, 2023.
[80] Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. A non-parametric test to detect data-copying in generative models. CoRR, abs/2004.05675, 2020.
[81] Marco Jiralerspong, Avishek Joey Bose, Ian Gemp, Chongli Qin, Yoram Bachrach, and Gau- thier Gidel. Feature likelihood score: Evaluating generalization of generative models using samples, 2023.
[82] Nicolas Pielawski and Carolina Wählby, Introducing hann windows for reducing edge-effects in patch-based image segmentation. PloS one, 15())ee022883, 2020.
16


Reviews:
[{"review_id": "vVfLBMvPkm", "reviewer": "Reviewer_LR7b", "summary": "", "questions": "", "limitations": "In an era of scientific reproducibility crisis where image faking is a common concern, I worry about the potential of misuse of this technology to create images that (wrongly) support particular biological and especially medical findings - it would be nice to have the authors discuss this.", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The ability to generate such images to the degree they can reliably fool experts is very impressive, especially across scales. A lot of work has been done to characterize the images, and given they come from medical data, I do especially appreciate the attention to dangers of memorization.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "R5WapjlQKz", "reviewer": "Reviewer_kGnb", "summary": "", "questions": "", "limitations": "The authors note the need for further optimizations in synthetic data generation process for some applications (like training deep learning models). ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "-\tThe method is able to produce large synthetic tissue images and their segmentation masks.\n-\tThe method includes an outpainting method that can be parallelized to speed up data generation process. \n-\tEvaluation with experienced pathologists as well as quantitative evaluation with fidelity and anti-copying metrics. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "DOZoiDWRF4", "reviewer": "Reviewer_Wamo", "summary": "", "questions": "", "limitations": "- Significant technical contributions rather than dataset/benchmark. The submitted work’s core focuses largely on technical contributions rather than the dataset itself. Also, It is unclear how the proposed framework can be eventually used to benchmark tasks i.e., what is the envisioned tasks?  \n\n- While the reviewer appreciates the extensive evaluations on quality generated images, another concern is how fair evaluations can be guaranteed if only the model is released, and not the generated images.  \n\n- Details regarding the downstream tasks are unclear [Section 5.3]. It is unclear whether an in-house or public dataset was used. In addition, while the trained model reaches 64.16 F1 score, it is equally unclear if this performance is significant as their is no mention of baselines or an upper-bound.  \n\n- Missing evaluation with DiffCollage (Table 1).", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper is well written and easy to follow. Also, the proposed framework is well motivated as synthetic image generation in medical data is challenging, especially in histopathology images that exhibit complex tissue micro-environments.  \n\n- While synthetic generation of images in digital pathology is not new; the proposed framework is able to generate arbitrarily sized images (up to 4096px), compared to related works that were limited to small resolutions and exhibit artefacts. At the same time, a small-scale dataset was used for model training.   \n\n- The sampling procedure introduced in this work is both novel and interesting. Also, efficient inpainting with different conditioning parameters is validated to generate diverse images.  \n\n- The authors extensively evaluated the generated images via a survey with experienced pathologists, including a downstream segmentation task and metrics for anti-duplications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BqUrkWbfAt", "reviewer": "Reviewer_hrbm", "summary": "", "questions": "", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* Whole Slide Images (WSIs) are overall quite costly to gather, and the community could greatly benefit from synthetic WSIs if they are created appropriately.\n* The sampling method they proposed seems to both avoid the artifacts resulting from tiling observed in earlier works (Zhang et. al. https://arxiv.org/abs/2303.17076), and it doesn't have the constraints that sliding sampling outpainting methods have, which can only be paralleled 2-4 times.\n* While it's highly risky to use patient data when training a generative model, it appears that the authors conducted experiments to address potential memorization issues. The generated samples seem to be diverse enough compared to the ones seen in the training data.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Strong contribution, and a well-written paper. It received all positive reviews. The paper is recommended for acceptance.
Decision: Accept (Spotlight)