CPSample: Classifier Protected Sampling for Guarding Training Data During Diffusion
Authors: Joshua Kazdan
Hao Sun
Jiaqi Han
Felix Petersen
Frederick Vu
Stefano Ermon
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion
privacy
data protection
mode collapse
Primary area: generative models
TL;DR: We provide a novel and adaptable way to prevent training data replication in diffusion models

Abstract: Diffusion models have a tendency to exactly replicate their training data, especially when trained on small datasets.  Most prior work has sought to mitigate this problem by imposing differential privacy constraints or masking parts of the training data, resulting in a notable substantial decrease in image quality. We present CPSample, a method that modifies the sampling process to prevent training data replication while preserving image quality. CPSample utilizes a classifier that is trained to overfit on random binary labels attached to the training data. CPSample then uses classifier guidance to steer the generation process away from the set of points that can be classified with high certainty, a set that includes the training data. CPSample achieves FID scores of 4.97 and 2.97 on CIFAR-10 and CelebA-64, respectively, without producing exact replicates of the training data.  Unlike prior methods intended to guard the training images, CPSample only requires training a classifier rather than retraining a diffusion model, which is computationally cheaper. Moreover, our technique provides diffusion models with greater robustness against membership inference attacks, wherein an adversary attempts to discern which images were in the model's training dataset. We show that CPSample behaves like a built-in rejection sampler, and we demonstrate its capabilities to prevent mode collapse in Stable Diffusion.

Published as a conference paper at ICLR 2025
CPSAMPLE: CLASSIFIER PROTECTED SAMPLING FOR GUARDING TRAINING DATA DURING DIFFUSION
Joshua Kazdan', Hao Sun', J Han?, Felix Petersen', Frederick Vu², Stefano Ermon
1 Department of Statistics, Stanford University
2 Department of Computer Science, Stanford University 3 Department of Mathematics, UCLA
ABSTRACT
Diffusion models have a tendency to exactly replicate their training data, especially when trained small datasets. Most prior work has sought mitigate this problem by imposing differential privacy constraints or masking parts of the training data, resulting in a notable substantial decrease in image quality. We present CPSample, a method that modifies the sampling process to prevent training data replication while preserving image quality. CPSample utilizes a classifier that is trained overfit on random binary labels attached to the training data. CPSample then uses classifier guidance to steer the generation process away from the set points that can be classified with high certainty, set that includes the training data. CPSample achieves FID scores of 4.97 and 2.97 on CIFAR-10 and CelebA-64, respectively, without producing exact replicates of the training data. Unlike prior methods intended to guard the training images, CPSample only requires training a classifier rather than retraining diffusion model, which is computationally cheaper. Moreover, our technique provides diffusion models with greater robustness against membership inference attacks, wherein adversary attempts to discern which images were in the model's training dataset. We show that CPSample behaves like a built-in rejection sampler, and we demonstrate its capabilities to prevent mode collapse in Stable Diffusion.
INTRODUCTION
Diffusion models are an emerging method of image generation that have surpassed GANs on many common benchmarks (Dhariwal & Nichol (021). achieving state-of-the-an FID scores on CIFAR- 10 (Krizhevsky) 2009). CelebA (Liu et al. 2015), ImageNet (Deng al.) 2009), and other touchstone datasets. Although their capabilities are impressive, diffusion models still suffer from the tendency to exactly replicate images found in their training sets (Carlini et al. 202 Jagielski et al. 2023 Somepalli et al.] 2023a). This problem is especially pronounced when the training set contains duplicates (Webster al. (023). Given that diffusion models are sometimes trained sensitive content, such as patient data KKazerouni et al.) 2023) Pinaya et al.) 2022) or copyrighted data (Dhariwal Nichol, 2021). this behavior generally unacceptable. Indeed, Google, Midjourney, and Stability Al are already facing lawsuits for using copyrighted data to train image generation models (Brittain) 2023 2024), 2024) some of which exactly replicate images from their training data during inference (Marcus
& Southen 2024).
The strongest formal guarantee against replicating or revealing training data is differential privacy (DP) (Dwork & Roth 2014). Unfortunately, differential privacy is at odds with generation quality. Although differentially private training has been implemented for GANs (DP-GAN) (Xie al. 2018), diffusion models (DPDM, DP-Diffusion) (Dockhorn et al., 2023 Ghalebikesabi et al. 2023), and latent diffusion models (DP-LDM) (Lyu et al. 2024), it typically results in significant degradation of image quality. Moreover, one cannot easily make pretrained model differentially private, implying that to achieve differential privacy, one must retrain from scratch. This makes negotiating the trade-off between privacy and quality challenging, trying different levels of privacy requires retraining. Due to the difficulty of achieving differential privacy while simultancously maintaining quality, some researchers have pursued more attainable model characteristies that have

---

Published as a conference paper at ICLR 2025
the same flavor differential privacy. A frequent benchmark for privacy is robustness to membership inference attacks (Hu & Pang 2023). whereby the attacker aims to infer whether given image was used to train the model. Although researchers have devised multitude of loss and likelihood-basee membership inference attacks, so far, there are few existing methods that explicitly aim to defend against these attacks besides differential privacy and data augmentation (Matsumoto al.) 2023b
Pang et al. 2023). A second privacy benchmark measures the cosine similarity in a feature space of generated image to its nearest neighbor in the training data (Daras al. 2024 Douze et al. 2024). Ambient diffusion (Daras et al. 2024) is one method to prevent excessive similarity to the training data without enforeing differential privacy; however, ambient diffusion still has notable negative
effects on FID scores.
Until recently, preventing image replication by diffusion models has involved various forms of data corruption during training, either by adding noise gradients (Abadi et al. 2016). diversifying images and captions (Somepalli ett al. 2023b), or corrupting the images them selves (Daras et al. 2024). Hyperparameter tuning for these methods requires retraining, making it difficult to calibrate them to the
necessary level of privacy. Simple alternatives, effective because
like rejection sampling, they can guarantee that the training images will not be exactly replicated. However, standard rejection sampling has major drawbacks, too. For instance, rejection sampling redistributes probability mass in an inefficient way and requires resampling, which can decrease speed. In extreme cases of mode collapse such as those uncovered by [Webster] (2023), Stable Diffusion must be queried dozens or even hundreds times before producing an original image. making rejection sampling impractical. Rejection sampling is also prone to membership inference attacks and privacy leakages (Awan & Rao 2023).
Figure I: Generated image and most similar train- ing image pairs for DDIM sampling (left) and CPSample with a=0.001, s=1 000 (right). We sample 100 images and display the four with the highest similarity to their nearest neighbors in the training data.
We present classifier-protected sampling (CPSample), diffusion-specine data protection technique that, while not strictly differentially private, fortifies against some membership inference attacks and greatly reduces excessive similarity between the training and generated data. CPSample is compu- tationally more efficient than existing training-based methods of improving privacy for diffusson models. The basic idea is to overfit classifier on random binary labels assigned to the training data and use this classifier during sampling to guide the images away from the training data. We show that our method has an effect similar to rejection sampling while removing or reducing the need to resample. Unlike rejection sampling, CPSample offers protection against some membership inference attacks during the generation process rather than only protecting the end product. CPSample achieves SOTA image quality, improving over previous data protection methods, such ambient diffusion, DPDMs, and PAC Privacy Preserving Diffusion Models (Xu et al. 2023) for similar levels of "privacy." Unlike most other methods designed to shield the training data, one can simply adjust the level of protection provided by CPSample without retraining the classifier used for guidance. CPSample is applicable existing image models without any expensive retraining of the diffusion models. We summarize the primary contributions of our work as follows:
we introduce CPSample, novel method of classifier-guidant for privacy protection in diffusion models that can be applied to existing models without retraining.
In Section 3.1
We show theoretically in Section 3.2 and empirically in Section 4.1 that CPSample prevents training data replication in unguided diffusion. We also provide evidence Section (.2tioonn.1ii that CPSample can protect text-based image generation models, like Stable Diffusion.
2

---

Published as a conference paper at ICLR 2025
We give empirical evidence that CPSample foil some membership inference attacks in Section [4.
We demonstrate in Section that CPSample attains better FID scores than existing methods of privacy protection while still eliminating replication of the training data.
2 BACKGROUND AND RELATED WORK
2.1 DIFFUSION MODELS
We begin with a review diffusion models. Denoising diffuusion probabilistic models (DDPMs) (Sohl Dickstein et al. 2015; Ho et al. (020)) gradually add Gaussian noise to image data during the "forward" process. Meanwhile, one trains "denoiser" to predict the original image from the corrupted samples in so-called "backward" process. During the forward process, one assigns
Zt VocTo + - o4c
(2.1)
where ~ (O I), Zo is the original image, and indicates the noise schedule. The variable t E {O..., } specifies the step of the forward process, where zy represents an image in the training data. When or is set sufficiently close to 0, 2T is approximately drawn from standard normal distribution. During intermediate steps, the distribution of xr
g ro) Nax;; voszo, (1 - Tt)
(2.2)
During training, one performs gradient descent on minimize the score-matching loss, given by
(o.
Σ
1 و 20
calvaroo + aie, t)|1
(2.3)
Here, D is the target distribution, which is approximated by sampling from the training data. Finally, to generate image, one samples standard Gaussian noise .T ~ (O, Then, one gradually denoises Vr by letting
1 Va,
1 az (0 (o(( VT-ri
Tt
31
+ o+Zt:
(2.4)
where in each step, one has 2n NO0 I). and o; and are scalar functions determined by the noise schedule that govern the rate the backward diffusion process.
Despite the superior image quality afforded by DDPMs, the sampling process sometimes involves 1 1000 or more steps, which has led to variety of sampling schemes and distillation methods for speeding up inference (Song al. 202T Kim et al. 2024 Song et al. 2023 Gu et al. 2023). One of the most commonly used modifications to the sampling process is denoising diffusion implicit models (DDIMs), which enable skipping steps in the backward process.
Currently, the state-of of- the-art art for guided generation is achieved by models with classifier- guid- ance (Ho & Salimans 2022). However, since CPSample employs classifier prevent replication of its training data, it is more useful for us to review its predecessor, classifier-guided diffusion (Lim et al. 2023 Dhariwal & Nichol 2021). In classifier guided diffusion, a pretrained classifier Pa(y xe, t) assigns probability the event that Dt Vairo /I are for some Tp with label y. The sampling process for classifier-guided DDIM is modified by
é = cx(x)) -
VI a. V 2, logpo xx.
(2.5)
F1
The
84 = 04-1
(2.6)
Va
Such modification of the sampling procedure corresponds sampling x, from the joint distribution:
p y t) = Zpo( Ze41, () x,, t)
(2.7)
where Z is a normalization constant. This formulation can be adapted for continuous-ting models, but for discrete-time models, additional care must be taken to ensure accuracy (see Appendix A for additional details).
3

---

Published as a conference paper at ICLR 2025
2.2 PRIVACY IN DIFFUSION MODELS
Differential privacy (DP) is generally considered to be the gold standard for protecting sensitive data.
The formal definition of (e-ô) differential privacy is as follows (Dwork al., 2006: Dwork & Roth 2014]:
Definition 2.1 ((e-))-Diffeed Diifferrent privacy). Let A be randomized algorithm that takes a dataset as imput and has its output in X. If D2 and D2 are datasets with symmetric difference 1, then A E-ô differentially private for all S C X. P(A(D)) E S) S. (((((2))) S)e² +
(2.8)
DP ensures that the removal or addition of a single data point to the dataset does not significantly affect the outcome of the algorithm, thus protecting the identity of individuals within the dataset. Existing DP diffusion models (Dockhorn et al.) 2023: Ghalebikesabi et al. 2023; Lyu et al., 2024)
achieve DP through DP stochastic gradient descent (DP-SGD) (Abadi et al. (016). which Gaussian noise is added to the gradients during training.
Though DP offers a formal guarantee that one's data is secure, imposing a DP constraint in practice severely compromises the quality of the synthetic images. Therefore, researchers have largely resorted to demonstrating that models exhibit various relaxations of strict DP (Vyas et al. 2023). such as
Probably Approximately Correct DP (PAC-DP) (Xiao & Devadas, (023) or other empirical metrics of privacy.
One common approach is to measure the similarity between generated image and its nearest neigh- bors in the training dataset with the aim of minimizing the number of highly similar neighbors (Daras) et al., 2022; 2024). Typically, similarity is quantified using either Euclidean distance or cosine similarit, which is given by
xf C(x)
(2.9)
||]||| ||C(=)]l
with C(z) denoting the nearest neighbor of among the training data, and which is typically computed in a feature space rather than the raw pixel space (Nguyen & Bai 2011 Xia et al. (015). Tools have been developed to help compute nearest neighbors efficiently, since naive pairwise comparisons too computationally expensive. In 2023, MetaAl developed the FAISS library for efficient similarity search using neural networks (Douze al. 2024), making this type of privacy metric possible to compute approximately in a reasonable amount of time. Empirically, for CIFAR-10, we observed that images with similarity scores above 0.97 were nearly identical, whereas for CelebA, the threshold was approximately 0.95, and for LSUN Church, images with similarity above 0.90 were sometimes, though not always, nearly identical.
Ambient diffusion reduces similarity to the nearest neighbor by masking pixels during training and only scoring the model based on the visible pixels. In this way, the model never has access to a full, uncorrupted image and is therefore less likely to replicate full images (Daras et al. 2024). The downside is that masking pixels, even at the relatively modest rate of 20%, leads to notable image quality degradation measured via the FID score. Moreover, ambient diffusion shifts the entire distribution of similarity scores towards lower similarity, whereas we should ideally prevent the generation of images with high similarity the training data while leaving the rest of the distribution untouched.
Until recently, all attempts enforcing privacy for diffusion models occurred during training. In 2023, Xu et al.] (2023) developed a method of classifier-guided sampling (PACPD) that has PAC privacy advantages over standard denoising. For text-guided models, (Somepalli al. (2023b) developed a method of randomly changing influential input tokens to avoid exact memorization, and |Wen| ct al.) (2024) protected training data using a regularization technique on the classifier-free guidance network during training. [oolatkar et al. (2024) used a mixture of public and private images to prevent copyright infringement, and Golatkar et al. (2023) developed "compositional diffusion models" to customize data access across different groups Recently|Chee et al.) (204)) devised guidance method (AMG) which calculates similarity metrics at each step in the denoising schedule in order to guide the sampling process away from similar data points in the training corpus. By utilizing similarity metrics directly, they were able to effectively eliminate memorization in both text-conditional and unconditional diffusion models. Though theoretically valuable, the need to have access to the training data- or at least to embeddings of the training data points- and to compute similarity measures at runtime is impractical for use outside of research environment.
4

---

Published as a conference paper at ICLR 2025
2.3 MEMBERSHIP INFERENCE ATTACKS
A third privacy measurement comes from membership inference attacks (Dubiński et al. 2024; Pang & Wang 2023 Duan et al. 2023 Wu et al. 2022), whereby one tries discern whether given data point was a member of the training set for the model. Robustness to membership inference attacks is implied by differential privacy. Membership inference attacks against diffusion models usually hinge on observed differences in reconstruction loss or likelihood that come from overfitting. In this paper, we will use slight modification of the membership inference attack from Matsumoto et al. (2023a) as described in Algorithm in Appendix H
3 PROTECTING PRIVACY DURING SAMPLING
In this section, we address the problem of training data replication in diffusion models, which poses significant privacy risks. One common solution to this problem is rejection sampling, whereby samples that closely resemble training data are discarded. However, rejection sampling has several shortcomings: it is computationally expensive and inefficient, and it only provides protection in the final output, not during the sampling process itself. Moreover, in extreme cases of mode collapse, one may need to generate dozens of images before generating original content when using rejection sampling.
To overcome these limitations, we introduce CPSample, a method that reproduces some of the benefits of rejection sampling without the need for resampling. CPSample integrates classifier guidance into the sampling process to steer the generation away from the training data. We overfit a classifier on random binary labels assigned to the training data and use this classifier during sampling to adjust the generated images, thereby reducing the likelihood of replicating training data while preserving image quality.
3.1 SAMPLING METHOD
The first step in CPSample is to train network that can provide information about how likely a sample xy is to turn into a member of the training data at the end of the denoising process. For this task, we classifier trained to memorize random binary labels assigned to the training data. It was shown in Zhang et al. (2117) that this can be achieved with network with a number of parameters that is linearly proportional to the size of the dataset, with a small constant of proportionality. Additionally, the training time required to memorize random labels is only small constant factor more compared to the time it takes to memorize real, non- non-random labels.
To address duplicated data in the training corpus, after the classifier has been sufficiently trained, items for which the classifier still shows significant loss can be reassigned a common label. Further training then ensures the classifier memorizes these items.
During the denoising process, whenever the classifier predicts a label y E {0, for xa with probability greater than 1 a, we perturb -z-1 towards the opposite label using classifier guidance. For example, if the classifier predicts the label with high probability, we employ classifier guidance to adjust the sampling process to draw from the conditional distribution pe, (3 x2, 4, y 0), reducing the likelihood of the generated sample being close to the training data.
To state our procedure more precisely, 1et ce(:s)) be the denoiser. Note that the classifier is trained only once on the training data and not during each sample generation. The sampling process is then modified in the following steps:
1. Randomly assign Bernuuli(0.5) labels to each member of the training data, and let B E {o, 13"" index these random labels. Train a classifier P. wt, t) predict these labels. Here, 2, is generated by corrupting the training data 20 with noise: a4 / Vaizo + vI are for ~ NO, I) and E (O...,, T)-
2. Set a tolerance threshold 0 < a < 0.5 and a scale parameter s. Let po | (1,() be the probability assigned the label y by the classifier p | [x2,)). Sample ør ~ N((0, I). For
E (T, ..... 1), if poly = 0 Iz, t) < a, replace eg (Ie, t) with
(0.0(11) 1) ca 1) SVI at V Vlog(po(u 0 x))
5

---

Published as a conference paper at ICLR 2025
If Ps t1,t) < a, replace Eg with
(0,0(22,)) t) co(2,, sv1 क. Vlog(p>(V = xx,()).
Otherwise, we leave the sampling process unchanged.
Though the choice of random labels for the classifier initially may seem counter-intuitive, it has several advantages over other approaches. If we used labels corresponding to real attributes of the data, the classifier would influence the content of the generated images in ways that could compromise their authenticity and diversity. This is because the guidance would push the generated images towards away from specific attributes, altering the intended distribution. The perturbation applied by the gradient of the log probability in CPSample moves the generated images away from regions where they can be easily classified similar the training data. This method is more effective than adding random noise, which would require a significant amount of noise to achieve the same effect, thus degrading image quality.
Unlike past training-based methods of privacy protection such as ambient diffusion and DPDM, once have trained the classifier, we can adjust the level of protection by tuning the hyperparameters and a without necessitating retraining of the classifier or denoiser. Our method also does not require access to the training data or excessive additional computation during sampling as the inferenced-basee method AMG does.
3.2 THEORY
In this section, we show that CPSample functions similarly to rejection sampling when preventing exact exact replication of the training images. We work under the following assumptions:
Assumption 1. Suppose that the classifier poly x. L) has Lipschitz constant L in the argument a with respect to a metric d(..-): X X + Rzo where x denotes the image space.
Assumption 2. Let be the random label assigned to I; E D, where D is the training data. Let be such that for all vi D. we have
Polys] x., 0) E (1 11--Ky k,
(3.1)
Assumption 3. Suppose that CPSample generates data &t such that X < poly 1) < 1
with probability greater than 1 - v, where we are able to govern v and X by adjusting and a in Section 3.7
In Assumption 1 the constant L can be difficult to evaluate, but the assumption holds for neural network classifiers Methods exist that can bound the local Lipschitz constant around the training data (Huang et al. 202I), which one can use to strengthen the guarantees of Lemmal [ Assumption holds well empirically, and Assumption 2 one can typically exert strong control over the size of without incurring too much additional computational overhead Zhang et al. (2017). Concretely, we were able to train our classifier to have cross-entropy loss below 0.05 in the experiments from Sections 4.1 and 4.2 Moreover, during sampling, observed that CPSample had control over the
quantity Po Tt. E). An example is given in Figure 7
demonstrate the following simple lemma, which links the behavior of CPSample to that of a rejection sampler without requiring expensive comparisons to the training dataset. A proof can be found in Appendix A We note that the assumptions strong, as Lemma designed primarily to give theoretical intuition for how CPSample works rather than practical guarantees how well it protects the data. We refer to reader to our empirical results (See Section 4 for practical demonstrations of the method's efficacy.
Given these assumptions,
admittedly relatively
Lemma Under the above assumptions, choose E > 0 and < < 1-xx Setting v = E and X = K+ L8. when drawing single sample, with probability greater than 1 1- CPSample generates an image that lies outside of 5 = ULED Bx(x) in the meiric space defined by d.
Note that the ability to control IP (i E U ULED Bx)x) gives the same guarantee offered by rejection sampling. However, in extreme instances of mode collapse such as those exhibited by Stable Diffusion in Section 4.2, one might have to resample hundreds of times to generate original images, making standard rejection sampling highly inefficient. CPSample able to produce original images without this high level of inefficiency.

---

Published as a conference paper at ICLR 2025
Similarty Scores with Nearest Neighbor for CIFAR-10
Similarity Scores with Nearest Neighbor for GelebA
1600
Standard DDIM
Standard DDIM
CPSample
CPSample
Number of samples
1200
Number of samples
60
800
40
400
20
8.5
0.7
0.8
1.0
0.5
0.6
0.7 0.8
0.9
s(x, NN(O)
s(x, NN(O)
Figure 2: Cosine similarity in feature space between generated images and their nearest neighbor in the annettunig dataset for standard DDIM sampling (red) and CPSample (blue) with a 0.001, s = on CIFAR- 10 (left) and with a = 0.001, s 000 on CelebA-64 (right). Similarity scores were com- puted for 21 000 generated samples for CIFAR- and 8 000 images for CelebA. Note that standard DDIM exhibits many more samples with similarity scores exceeding the thresholds from Table [
4 EMPIRICAL RESULTS
We three distinct sets of experiments demonstrate the ways in which CPSample protects the privacy of the training data. First, we statistically test the ability of CPSample to reduce similarity between generated data and the training set for unguided diffusion. We then demonstrate that CPSample can prevent Stable Diffusion from generating memorized images. Finally, we measure robustness against membership inference attacks. Hyperparameterss in all empirical tests, are chosen to maximize image quality while eliminating exact matches. In our tests of image quality, we find that CPSample far outperforms existing methods of protecting the training data.
4.1 SIMILARITY REDUCTION
We generate images using DDIM with CPSample and 0.0 denoising steps. The nearest neighbor to each generated image is found using Meta's FAISS model (Douze et al. 2024). Similarity between two images is measured by cosine similarity in a feature space defined by FAISS. We empirically find that similarity score exceeding 0.97 often indicates nearly-identical images for CIFAR-IO. For CelebA and LSUN Church, the thresholds lie around 0.95 (Daras et al. 2024) and 0.90, respectively. necessary but not sufficient
Note that a cosine similarity score above the thresholds given condition for images to look very alike. To ensure that we can observe a larger number of images with similarities exceeding our thresholds, we fine-tune the models using DDIM (Song et al.) (02T]) on subset of the data that consisted of 1 000 images, was done in Daras et al. (024). This modification allows us to statistically test the efficacy of CPSample without the large number samples required to do hypothesis testing rare exact replication events. After fine-tuning, up to 12.5% of the images produced by unprotected DDIM are nearly exact replicates the fine-tuning data. One can see from Table T that CPSample significantly reduces the fraction of generated images that have high cosine similarity to members the fine- tinettnnin set. One can see histograms of the similarity score distribution with and without CPSample in Figures and TO Figures 1 and show the most similar pairs of samples and fine-tuning data points. Uncurated images generated from CPSample can be found in Appendix [F] While CPSample effectively reduces the similarity between generated images and the training data, our results in Table dfeeeicidicet that CPSample achieves minimal degradation in quality compared to previous methods.
4.2 STABLE DIFFUSION
As a second demonstration of CPSample, we present evidence that CPSample can prevent well-known examples of mode collapse in near-verbatim attacks against Stable Diffusion (Webster 2023 Wen et al. 2024). We curate a small dataset of commonly reproduced images (Somepalli et al. (023b) and include other images from the LAION dataset depicting the same subjects, while ensuring that this dataset contains no duplicates. In this more targeted application, CPSample can prevent exact replication when used with the right hyperparameters. See Figure 6 and Table for more details.
7

---

Published as a conference paper at ICLR 2025
DDIM (Unprotected)
CPSample (Protected)
DDIM (Unprotected)
CPSample (Protected)
Figure 3: Generated images and their most similar training image pairs for DDIM sampling and CPSample with a 0.001, s 1 on CIFAR-10 (left) and 0.1, 8 10 LSUN Church (right). For each pair, the image on the left the generated sample, and the one on the right is its nearest neighbor in the training set. These are the four examples out of 21 000 images on CIFAR-IO and two out of 700 images on LSUN Church with the highest similarity scores with their nearest neighbor.
Table 1: Reduction in cosine similarity between generated images and nearest neighhor in fine-tuning data. p-valuus were computed using a x² test for the null hypothesis No: "CPSample did not reduce the fraction of images with similarity score exceeding the threshold."
Dataset
FT Steps
a
Scale
Threshold
DDIM
CPSample
p-value' 1
CIFAR-10 CelchA
150k
0.001
0.97
6.25%
0.00 %
<0.0001
650k
0.001
1 000
0.95
12.5%
0.10%
<0.0001
LSUN Church
455k
0.1
10
0.90
0.73%
0.04%
0.013
Although CPSample does not provide as robust protection in this setting compared to Somepalli et al. (2023b): Wenn et al. (2024), these results highlight its potential for data protection in text- diffusion models. Morcover, the methods developed in Somepalli et al. (2023b): Wenn et al. (2024) do
not apply unguided diffusion models. This application also highlights how CPSample only a private subset the training data, for which it does not require access to the entire training dataset.
protect
Table Details of generation on Stable Diffusion.
Image
Original caption
Modified caption
scale
guidance
A
"Rambo and Rocky Spin-Oft Off
"Rocky and Rambo Spin-Off- -
0.5
2 000
1.5
Sylvester Stallone gibt Updates"
Sylvester Stallone gibt Updates"
B
"Classic cars for sale"
"Classic for sale"
0.3
100
1.5
C
"Red Exotic Fractal Pattern Abstract Art On Canvas-7 Panels"
"Red Exotic Fractal Pattern Abstract Art On Canvas-7 Panels"
0.5
2 2000
1.5
4.3 MEMBERSHIP INFERENCE ATTACKS
We also assess CPSample's ' ability to protect against membership inference attacks. Following Algorithm [ we compute the mean reconstruction error for the training and test datasets and determine whether there is a statistically significant difference. To evaluate resistance to inference attacks, we use model trained on the entire set 50 000 CIFAR-I0 training images. We compare the reconstruction loss these 50 000 training images to the reconstruction loss on the 10 withheld test samples included in the CIFAR-10 10 dataset. We compare the difference in reconstruction loss between these two datasets both for CPSample. using classifier trained on the entirety of the CIFAR- 10 training data with random labels, and for standard DDIM sampling. We demonstrate CPSample's 'ss resistance to inference attacks for {0.5, 0.25, 0.001) over approximately 8000
8

---

Published as a conference paper at ICLR 2025
images from each of the training and test datasets. The p-values in this experiment based two-sample, single-tailed Z-score that tests the null hypothesis "the average training reconstruction loss is less than or equal to the average test reconstruction loss." Precisely, let denote the number of training data points and m denote the number of sampled test data points. The test statistic is then given by
Prest
Mtrain
Viees/ m Vuain/
Here, the symbol V indicates the variance and indicates the mean. In this context, failure to reject the null hypothesis indicates success for CPSample.
We observe that in experiments, very low value of leads to higher p-value, which is counter- intuitive on first glance. However, we believe that this occurs due to the fact that a small value of results in more targeted application of CPSample, driving the loss up exclusively around the training data points. As shown in Table for values a between O and 0.5, a conclusive membership inference attack against CPSample is not possible. We provide second black-box membership inference attack based on permutation testing in Appendix
Figure 4: Uncurated samples from DP-LDM (yu et al. 2024) on CelebA (Top) and Ambient Diffusion (Daras et al. 2024) with corruption 0.2 (Bottom). These samples of noticably lower quality than the uncurated samples from CPSample, found in Appendix
Figure 5: The generated and real im- ages with the highest similarity for CIFAR- 10 (left) and CelebA (right) out of 50 000 samples used to com- pute FID score.
4.4 QUALITY COMPARISON
As mentioned in the introduction and Sec- tion 4. other methods privacy protection A suffer from severe degradation of quality mea- sured by FID score. Here, we provide an FID score comparison between the CPSample model fine-tuned on curated subsets of CIFAR-10 and CelebA and existing methods of privacy protec- tion. FID scores for unconditional generation of CIFAR-10 and CelebA are presented in Table
The images with the highest similarity to the training set. determined using FAISS, are shown
in Figure 5 The particular values of a and
were set an attempt to find the least aggres- sive settings that still completely prevent exact replication of the training data. FID scores over grid search on the hyperparameters and
are displayed Table 7 We include images from DP-LDM (Lyu et al. 2024) and Ambient Diffusion (Daras et al.) 2024) in Figure 4 to em-
Figure 6: Selected examples for Stable Diffusion: original image (left), image generated from a simi- lar caption by Stable Diffusion v1.4 (center), image generated with CPSample (right).
phasize the gains in quality.

---

Published as a conference paper at ICLR 2025
Table 3: Difference in mean reconstruction error between train and test data for CIFAR-10.
Table 4: FID score comparison on the CIFAR- 10 and CelebA datasets.
Method
CIFAR-10 CelebA
Method
Test statistic p-value
DDIM
3.17
1.27
DDIM
138
Rs
Ambient (Corruption 0.2)
0,141
0.44
Ambient (Corruption 0.2)
11.70
25.95
Ambient (Corruption 0.8)
-0.024
0.51
DPDM 10)
97.7
78.3
DP-Diffusion (E 10)
9.8
CPSample (a 0.5)
0.59
0.28
DP-LDM (e 10)
8.4
16.2
CPSample (a 0.25)
0.23
0.41
CPSample (a = 0.001)
-0.86
0.81
CPSample (a 0.001, 0.05)
4.97
2.97
5 LIMITATIONS
As mentioned Section 3.1 the difference training time required to get a classifier to memorize random labels versus real labels has been shown to be only small constant factor Zhang al. (2017). Compared to other leading methods of protecting training data, such as ambient diffusion, DPDM, and AMG, our method is significantly casier to employ in terms of computational resources. However, as we lack the resources to provide further empirical evidence beyond what has already been demonstrated in the literature, leave this remark flag for potential practical limitation of our method.
Of slight theoretical concern is the difficulty in providing practical upper bounds the Lipschitz constant of the classifier, for which a lower value would provide stronger formal guarantees of privacy, Further research into employing Lipschitz regularizations may both improve the performance of our method and provide stronger guarantees. In practice, we observe stronger protections than the formal guarantees provide. Thus, we include the formal guarantees primarily as a source of intuition.
CONCLUSION
We have presented new approach to prevent memorized images from appearing during inference time. Our method is applicable to both guided and unguided diffusion models. Unlike previous methods intended to protect privacy of unguided diffusion models, CPSample does not necessitate retraining the denoiser. Moreover, the presence of duplicated data in the training corpus does not affect on our approach, and after training the classifier, one can adjust the level of protection enforced by CPSample without further training. We have shown theoretically that method behaves similarly to rejection sampling without necessitating resampling. Finally, we have provided empirical evidence with rigorous statistical testing that our method effective in unguided settings. We have also given examples in which CPSample was able to prevent extreme instances of mode collapse in Stable Diffusion. Despite its efficacy at preventing replication of training images, CPSample has little negative impact on image quality.
REPRODUCIBILITY STATEMENT
We provide source code, scripts, and configuration details for experiments in the supplementary material for those seeking reproduce this study. Proofs of original claims are given in the appendix, along with details for the implementation and training of the model. Statistical measures of significance are included ensure the robustness our results.
ETHICS STATEMENT
This paper addresses privacy concerns generative AI models. Our method designed to mitigate these risks by safe guarding against membership inference attacks and data replication. We caution users that although our method prevents exact replication training data, the theoretical assumptions relaxed in practice, so it does not guarantee complete protection. We encourage others to act responsibly in the application this work and to further research methods to prevent exact replication of trainign data.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '16, pp. 308-318, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450341394 doi: 10.1145/2976749.
2978318. URL httph:tp/////.m. : doi. Pooi.org//00 1145229777799 2978318
Jordan Awan and Vinayak Rao. Privacy-aware rejection sampling. Journal of machine learning research, 24(74):1-32, 2023.
Blake Brittain. Artists take new shot at stability, midjourney in updated copyright law-
suit. Reuters, 2023. URL http://wwww.eeetttts.coaaaaaaa/a
google-sued-bye Setaadbyyyu--aiisss-----o- overrss--ver--oo ai ai-aaaii-ieeeeaa image- 2atpaaaeeegeeertttto-222------ 2024-04-292/-
Blake Brittain.
Google sued by us artists over ai image generator. Reuters,
2024.
http://www.rrttes.ccooooaa/aaaaa goog le-leel-dd org.
URL
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee. Adam Roberts, Tom Brown, Dawn Song. Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Sympo- sium (USENIX Security 2/), pp. 2633-2650 USENIX Association, August 2021. ISBN 978-1 wtw.e//www.....cooooeeeeeeeeeeaa http://///ssssxeeeeeeettxyl
939133-24-3. URL htwp::ttt...... http://wwwwwwwniixe..e
Dresentatiioncarllinn
Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in superviid vision transformers, 2021.
Chen Chen, Daochang Liu, and Chang Xu. Towards memorization free diffusion models, 2024. URL lattps://arxte org/ abs/ 2404.4444 00922
Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G. Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions, 2022.
Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Am- bient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems, 36, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255 2009. URL http::
Imagenet_cvpr033 pdd.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Tim Dockhorn, Tianshi Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffusion
models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https
//openreview.net http://wwwwi.eeeeettn.....a wttp://mmmmmmmpp11m111
Matthijs Douze, Alexandr Guzhva, Chengqi Deng. Jeff Johnson, Gergely Szilvasy, Berre-Emmmnuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library, 2024.
Jinhao Duan, Fei Kong, Shiqi Wang, Xiaoshuang Shi, and Kaidi Xu. Are diffusion models vulnerable to membership inference attacks?, 2023.
Jan Dubiáski, Antoni Kowalczuk, Stanislaw Pawlak, Przemyslaw Rokita, Tomasz Trzciáski, and Pawel Morawiccki. Towards more realistic membership inference attacks on large diffusion models. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), 01 2024.
Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science, 2014.

---

Published as a conference paper at ICLR 2025
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. volume Vol. 3876, pp. 265-284, 01 2006. ISBN 978-3-540-32731- doi: (0.1007/1168187 14.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- weighted linear units for neural network function approximation in reinforcement learning. 2017.
Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Sofia Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate useful synthetic images, 02 2023.
Aditya Golatkar, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Training data protection with compositional diffusion models. CoRR, abs/2308.01937, 2023. URL https
48550/85588/ 001.i/i..300..1130 01937
Aditya Golatkar, Alessandro Achille, Luca Zancato, Yu-Xiang Wang, Ashwin Swaminathan, and Ste- fano Soatto. CPR: retrieval augmented generation for copyright protection. CoRR, abs/2403.18920, 2024. / 24922......0
Jiatao Gu Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Josh Susskind. Boot: Data-free distillation of denoising diffusion models with bootstrapping, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Hailong Hu and Jun Pang. Membership inference of diffusion models, 2023.
Yujia Huang, Huan Zhang. Yuanyuan Shi, Zico Kolter, and Anima Anandkumar. Training certifiably robust neural networks with efficient local lipschitz bounds. In A. Beygelzimer, Y. Dauphin, P. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. BRLpttps: http///wpppp....pp////p/p. net Fttp:/Eorrnnaa--EF222..2.2
Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito. Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, and Chiyuan Zhang, Measuring forgetting of memorized training examples. The Eleventh International Confer- ence on Learning Representations, 2023. Wttphttp::///t htttps::/openreveee..o net. http//wwwwee/.rrrreneeee forum? id=
7bJizxLKrR
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models, 2022.
Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, llker Hacihaliloglu, and Dorit Merhof. Diffusion models for medical image analysis: A compre- hensive survey, 2023.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning prob- ability flow ODE trajectory of diffusion. The Twelfth International Conference on Learning Representations, 2024. https
net/ Fet:/oormmidddyajja... TTDDD
Alex Krizhevsky, Learning Multiple Layers Features from Tiny Images. Technical report, Univer- sity of Toronto, 2009. URL https:// http://wwwwc cs toronto. edt/=krii/ cifar.iaaaa.// himl
Sungbin Lim, Eunbi Yoon, Taehyun Byun, Taewon Kang, Seungwoo Kim, Kyungjae Lee, and Sungjoon Choi. Score-based generative modeling through stochastic evolution equations in hilbert spaces. In seveth Conference on Neural Information Processing Systems, 2023. URL hittps://openreet
net forum? did= RvXnF
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. Proceedings of International Conference Computer Vision (ICCV), 2015.
12

---

Published as a conference paper at ICLR 2025
Saiyue Lyu, Michael F. Liu, Margarita Vinaroz, and Mijung Park. Differentially private latent
diffusion models, 2024.
Gary Marcus and Reid Southen. Generative ai has a visual plagiarism problem, 2024. URL https:// /ttp::/spctttmmmmm ieee. wwt.eeerggmmmmmouomaooe copyrightt
Tomoya Matsumoto, Takayuki Miura, and Naoto Yanai. Membership inference attacks against diffusion models, 2023a.
Tomoya Matsumoto, Takayuki Miura, and Naoto Yanai. Membership inference attacks against diffusion models. pp. 77-83, 05 2023b. doi: 10. 10099//...933..223...00
Hieu V. Nguyen and Li Bai. Cosine similarity metric learning for face verification. In Ron Kimmel, Reinhard Klette, and Akihiro Sugimoto (eds.). Computer Vision ACCV 2010, pp. 709-720, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg, ISBN 978-3-642-19309-5.
Yan Pang and Tianhao Wang. Black-box membership inference attacks against fine-tuned diffusion models, 2023.
Yan Pang, Tianhao Wang. Xuhui Kang, Mengdi Huai, and Yang Zhang. White-box membership inference attacks against diffusion models, 2023.
Walter H. L. Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M. Jorge Cardoso. Brain imaging generation with latent diffusion models, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
Jascha Sohl- Sicl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265 PMLR, 2015.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art digital forgery? investigating data replication in diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048-6058, 2023a.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Under- standing and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783-478880 2023b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Inter- national Conference on Leurning Representations, 2021. httt:httt////eeeeeeet
net/
Eorum?
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023.
Nikhil Vyas, Sham M. Kakade, and Boaz Barak. On provable copyright protection for generative models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 35277-35299, PMLR, 23-29 Jul 2023. URL htt.hhttts//ppceeeeeeeee ngs. p
html
Ryan Webster. A reproducible extraction training images from diffusion models, 2023.
Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b. 03 2023.
Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memo- rization in diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL
net Forum?iee 30wkH
Yixin Wu, Ning Yu, Zheng Li, Michael Backes, and Yang Zhang. Membership inference attacks against text-to-image generation models, 2022.
13

---

Published as a conference paper at ICLR 2025
Peipei Xia, Li Zhang, and Fanzhang Li. Learning similarity with cosine similarity ensemble. Information Sciences, 307:39-55 52, 2015. ISSN 0020-0255. doi: https://doi.rr//10.0066// 2015.02.024. URL https:/ www.
2002002551500124
Hanshen Xiao and Srinivas Devadas. Pac privacy: Automatic privacy measurement and control of data processing. In Anmual International Cryptology Conference, pp. 611-644. Springer, 2023.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network, 2018.
Qipan Xu, Youlong Ding, Jie Gao, and Hao Wang. Pac privacy preserving diffusion models, 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017 Conference Track Proceedings. OpenReview.net, 2017. URL https: http:::///peeewwww..roo net / for um ? id Sy
14

---

Published as a conference paper at ICLR 2025
A PROOFS
Details of classifier guidance For completeness, we include derivation of the classifier-guidane introduced in (Dhariwal& & Nichol 2021).
During the conditional denoising process, should sample 22-1 from the conditional distribution
P (et v) (P(x--1,(e.1) P 3t P(y (r-21-11
(A.1)
P((,,,)) v)
PP(y | xx))
One can show that Pyy Tie xt-1) Pfyy 14-1) (see Dhariwal & Nichol (2021 for details). The denominator Ply X4)) does not depend on 2t-1- Therefore, we write this term as Z. To get an
estimate of the probability P(y | Tz-1), we train classifier of the form po xx-1)- Thus, we should estimate the conditional probability IP(x;-1 Zt. y) via
(2.( = (po xe)pp(y 2t-1).
(A.2)
In continuous time, we can write p( v) = p(xx)p(( æz), and the score function is:
au (og(po(x:)24 xx)) = Zz, log + Vz, log at)-
(A.3)
The network c0(2, () predicts the noise added to sample, which can be used to derive the score function
V se Logpe((rr,
VI a
Substituting this into equation
A.3 we get
(x(()) Va, logpo 2z).
(A.4)
at
This leads to a new prediction for
(Φ(x;) = co (x,)
v1
T. Ve, log xx)-
The conditional sampling then follows in the same manner as standard DDIM with replaced by ég.
Proof of Lemma
Proof. Let x' E Bs(zo), where 20 E D is assigned the random label y. By Lipschitz continuity, have that
1p æo. t) po(y a < Ld(ro, 2').
By Assumption
2 we have (3 zo, 0)
K, follows that
p x'.0) = (
Zo, 0) Poly To. 0) + ps ( zz,0)
Po (y
To. 0) (po(y) Fo. 0) - Po x' 0))
2 Po (y To 0) Poly xo. 0) ( z',0)l
2 ( Fo. 0) Ld(xo, x')
2 (1) Fo, 0) L5
L8
X.
- 1, By Assumption 3 Therefore, for all points x' S, have poly x2,0) [0. CPSample generates samples with (2 2) E 1A,1-X] with probability at least E. Thus, we have that CPSample generates samples outside of S with probability least 1
15

---

Published as a conference paper at ICLR 2025
0.65
0.60
Predicted probability
0.55
0.45
SD
CPSample
10
20
30
40
50
Figure 7: CPSample is able to generate images with Po(y &. 0) (A, A). This example shows the probability Pyly = xr. () during the generation process with Stable Diffusion guided by the caption "Rambo and Rocky Spin-Off Sylvester Stallone gibt Updates." Note that higher step indicates later point in the denoising process. In this example, Stable diffusion exactly replicated the memorized image of Stallone, whereas CPSample (a 0.5, s 2000) produced original image.
B CLASS GUIDED DIFFUSION
As final experiment, we implement CPSample alongside classifier-free guidance for CIFAR-10 to ensure that CPSample does not cause frequent out-of-category samples. The models used for guided diffusion were smaller, so the image quality is naturally lower.
Figure 8: Uncurated samples using classifier-fre free guidance on CIFAR-10. 10. The image in the position second row, third column from the top left is near-exact replica of member of the training data.
16

---

Published as a conference paper at ICLR 2025
Figure 9: Uncurated samples using CPSample (e 0.1, 10) along with classifier-fre free guidance on CIFAR-10. Note that although CPSample slightly reduces image quality, it does not cause out-of- category samples.
C TRAINING DETAILS
Training classifiers. For training the classifier, we randomly selected subsets of 1 000 images each from the CIFAR-10, CelebA, and LSUN Church datasets, on which we trained the classifier from scratch. The architecture of our classifier is modified version of the U-Net model. We retained key components of the U-Net (Ronneberger et al. 2015) model structure, including the timestep embedding, multiple convolutional layers for downsampling. and middle blocks. The output from the middle blocks underwent processing through Group Normalization, SiLU (Elfwing al. 2017)) activation layers, and pooling layers before being fed into a single convolutional layer, yielding the classifier's output. Parameters for layers identical to the standard U-Net were consistent with those used to pretrain the DDIM model on these datasets. Additionally, akin to the pretraining of DDIM, incorporated Exponential Moving Average during training stabilize the training process. The training of each classifier model was conducted using 4 NVIDIA A4000 GPUs with 16GB of memory. For subsets of 1 000 images, the classifier took only hours train. For larger datasets consisting of 60 000 160 000 data points, the classifier took up to I week to train. By comparison, retraining diffusion model to be differentially private or using the method presented (Daras et al. 2024) can take weeks or months depending on the dataset.
Fine-tuning pretrained denoiser model on subsets. For fine-tuning the pretrained denoiser model on subsets, we commenced with the 500 000-step pretrained checkpoints available for the denoiser DDIM model. Fine-tuning was performed on subsets of 000 images each from the CIFAR- 10, CelebA, and LSUN Church datasets until the model began generating data highly resembling the respective subsets. The number of training steps varied across different models, and specific details regarding the fine-tuning process can be found in Table 5 Throughout the fine-tuning process, hyperparameters remained consistent with those used during the pretraining phase. We employed NVIDIA A5000 GPUs with 24GB of memory for fine-tuning each model the subsets.
D EVALUATION DETAILS
Numerical stability For the purposes of numerical stability, slightly modified the sampling process described in Section 3.1] We noticed in earlier iterations of our method that very small numbers of images were becoming discolored or black because in float16, the classifier predicting
17

---

Published as a conference paper at ICLR 2025
Table 5: Training Parameters Steps
Batch Size
LR Optimizer EMA Rate Classifier Steps Fine-tune Steps
CIFAR-10
256
2e-4
Adam
0,9999
560 000
000
CelebA
128
2e-4
Adam
0.9999
610.000
150
LSUN Church
8
2c-5
Adam
0,999
1 250 000
880
probabilities of 0.0000 or 1.0000 for the random label causing the logarithm to blow up. To fix this in practice, we do the following. Sample øT NN, f). For E {T, ..... 1}, if P = 0/z,, t) < a, replace (2 1) with
fg.0 1) = c0( 1) SVT Te Vlog(T + po(y 0f24,
If poly = 1/r,, 1) a, replace Eg with
(z,, t) c0(x t) sVI - a.
Vlog(T + Po 1(2,,
Otherwise, leave the sampling process unchanged.
By setting equal to 0.001, we were able to prevent the undesirable behavior.
Similarity Reduction Evaluation. We employ the fine-tuned denoiser model to generate 000 image samples for each of the aforementioned datasets. Additionally, utilize the Classifier-guided method to generate another set of 000 images. Subsequently, we employ DINO (Caron et al.) 2021) to find nearest neighbors in the subset using methodology akin ambient diffusion. From the perspectives of both DINO's similarity scores and human evaluation, we observe that images generated through the classifier guided approach exhibit significantly lower similarity the original images in the subset compared to those generated without guidance.
FID Evaluation. For each dataset, we utilize the denoiser model fine-tuned on the subset to generate 30 000 images under the guidance of the classifier. Subsequently, we employ the FID score implementation from the EDM (Karras et al. 2022) paper to compute the FID score.
goal of method, provide some context for how fast is compared to standard diffusion. We do our comparison using a batch size of to generate 10 images with 50 denoising steps. CPSample with 0.5 (i.e. computing gradients of the classifier every step) had an average per-image generation time of 26.1 + 0.029s. By contrast, standard stable diffusion had average generation time of 23.92 + 0.055s. Therefore, when the classifier small compared to the size of the diffusion model, the added time cost is insignificant.
Inference Speed Although speed
E
MEMBERSHIP INFERENCE ATTACKS
Algorithm Test statistie for membership inference attack against diffusion models (Matsumoto et al. 2023a)
Input: Target samples Ti, Im. CPSample denoiser ég,g. noise schedule ā. = II (1 B.) total_error error +
for in (#1,..., Fm do total_error +
+ ||c Co,6(VaT + VI -ic, f)12
end for mean_error <- total_errop rotal_error/m.
keeping with our goal preventing membership inference attacks that are based on high similarity to single member of the training set, we also perform permutation test to ensure that are not producing images that are anomalously close to the training data. Explicitly, we test the null hypothesis: generating images from CPSample produces images that training data than they are to arbitrary points drawn from the data distribution. Our tests are performed in the same setting used in Section 4.1 Let S (.1,-..,;2) be the data used for fine-tuning. Let
more similar to the
18

---

Published as a conference paper at ICLR 2025
T {Ii, ......... Tk Dk+1, .... ..., Fnl be the entire training set. Finally, let P {01, .... Zm be samples from CPSample. Then our permutation test is as follows:
1. Sample 1,...., Ty from P without replacement. For each ē, compute the quantity in 2.9 where the nearest neighbor chosen among S. Let the similarity score of the most similar pair be a.
2. Repeat the following process times: sample S C T without replacement from I so that |SS| k. Sample P without replacement from P so that [P = k. Compute the most similar image S for each member of Pt, Call similarity of the most similar pair a,
3. For a pre-specified level at, reject the null hypothesis
Ei= i= 1a00 > 0;} > a.
The results can be found in Table 6 Note that the test fails to reject CIFAR-I0 and LSUN Church, succeeds on CelebA. This likely because we fine-tuned the CelebA model more extensively than the other two.
Table 6: Reduction in cosine similarity between generated images and nearest neighbor in fine-tuning data.
Dataset
FT Steps
Scale
DDIM
CPSample
CIFAR-I0
150k
0.001
0.92
0.47
CelebA
650k
0.001
1000
0.99
0.99
LSUN Church
455k
0.1
10
0.99
0.60
I p-values were computed using x² test for Ho: CPSample did
not reduce the fraction of images with similarity score exceeding the threshold.
F ADDITIONAL EMPIRICAL RESULTS
Table 7: FID score w.r.t a and Scale on CIFAR-10.
0.001
a 0.01
a 0.1
0.25
0.49
Scale 4.14275
4.15467
4.19058
4.19208
4.21859
Scale
4.15772
4.20731
4.36005
4.58839
4.9566
Scale
4.18083
4.26594
5.05858
6.17326
7.88949
Scale 100
4.96727
16.7173
74.7247
113.199
139.626
Similarity Scores with Nearest Neighbor for LSUN Church
70
Standard DDIM CPSample
Number of samples
40
20
10
0.0
0.2
0.8
1.0
s(x. NN(x))
Figure 10: Similarity scores with nearest neighbor for standard DDIM and CPSample (a 0.1. scale= 10) on LSUN Church. In both cases, the network was fine-tuned for 455k gradient steps on subset of 000 images.
19

---

Published as a conference paper at ICLR 2025
H
Figure 11: Uncurated samples using standard DDIM fine-tuned for 455k gradient steps on subset of 1 000 images from LSUN Church.
20

---

Published as a conference paper at ICLR 2025
Figure 12:
Uncurated samples using CPSample (a 0.1, scale= 10) applied to a network fine-tuned for 455k gradient steps subset of 1 000 images from LSUN Church. Note that there is visual discrepancy in quality between these and the images from standard DDIM.
21

---

Published as a conference paper at ICLR 2025
Figure 13: Uncurated samples using standard DDIM fine-tuned for 580k gradient steps on subset of
1 000 images from CelebA.
Figure 14: Uncurated samples using CPSample (a 0.001, scale= 1 1000 applied to network fine-tuned for 580k gradient steps on subset of 000 images from CelebA. Note that there little visual discrepancy in quality between these and the images from standard DDIM.
22

---

Published as a conference paper at ICLR 2025
Figure 15: Uncurated samples using standard DDIM fine-tuned for 150k gradient steps subset of 1 000 images from CIFAR-10.
Uncurated samples using CPSample (a = 0.001, scale= 1) applied to network fine- tuned for approximately 150k gradient steps subset of 1 000 images from CelebA. Note that there is little visual discrepancy in quality between these and the images from standard DDIM.
Figure 16:
23


Reviews:
[{"review_id": "HjIH1Costw", "reviewer": "Reviewer_6foL", "summary": "The paper proposes Classifier-Protected Sampling (CPSample), a novel technique to prevent diffusion models from generating training data  duplicates, apart from previous methods such as rejection sampling or other sampling methods that occurs during training phase, CPSample allows the diffusion model to steer away from generating training samples during test time, resolving the problem of privacy leakage.", "questions": "1. As mentioned in the first point of weaknesses, it would be great if the authors could elaborate more on the the third assumptions either empirically or theoretically along with the $\\delta$ problem mentioned.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method presents interesting facet and novelty where a classifier is utilized in remembering the statistics of training distribution and acts as guidance to steer away from the training data.\n2. The planned experiments are great in terms of studying the effect of CPSample in a holistic manner. The three different experiments demonstrate different aspect of CPSample which is helpful for understanding the empirical effect.", "weaknesses": "1. The presented theoretical study seems to be not proven much in terms of CPSample. Specifically, it is not demonstrated as to why assumption 3 holds well empirically as stated in the paper. Furthermore, given the intractable estimation and sometimes large values of Lipschitz constant, I wonder if that would make the possible $\\delta$ extremely small which in terms renders the theory results in vain?\n2. Some more experiments on MIA could be carried out to effectively evaluated the information loss caused when CPSample. Particularly, papers such as [1, 2] have devised new MIAs tailored to the settings of diffusion models which also conforms to the past settings in MIA.  It would be great to see if CPSample could also withstand these newly devised attacks.\n\n\n\n[1]: Duan, Jinhao, et al. \"Are diffusion models vulnerable to membership inference attacks?.\" International Conference on Machine Learning. PMLR, 2023.\n[2]: Kong, Fei, et al. \"An efficient membership inference attack for the diffusion model by proximal initialization.\" arXiv preprint arXiv:2305.18355 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZWCjGQJ0uY", "reviewer": "Reviewer_8m3n", "summary": "The paper proposes a method to prevent the generation of training data during the inference on diffusion models. The way to obtain the result is by modifying the sampling process in diffusion models. The method leverages a classifier that is trained and forced to overfit on random binary labels assigned to the training data of diffusion models. The method uses “classifier guidance” to prevent the generation process from returning the training data points.", "questions": "1.\tThe abstract states: “CPSample achieves FID scores of 4.97 and 2.97 on CIFAR-10 and CelebA-64, respectively, without producing exact replicates of the training data.” What is the baseline? What model was it checked on? Without this background and some reference, these results do not have any meaning.\n2.\tThe paper should cite another work that applied DP to diffusion models [1].\n3.\tIntroduction: this method “fortifies against some membership inference attacks” This statement is rather too vague and it should be clearly stated what the goal of the method is. The defenses should not prevent some attacks that can infer the training images because the adversary would simply use the attacks that work and are not prevented by the proposed method.\n4.\tThe fact that better FID scores are achieved than other methods has to be presented with some reference. Is it the trade-off that the higher performance stems simply from the worse protection?\n5.\tThe background of the paper takes 2 pages. It is way too long and should be limited to a single page. For example, DP is not used in this paper so there is no need to provide this background section. Moreover, providing the formula for cosine similarity is really too trivial. The statement about the empirical results from this paper in Section 2.2 is very surprising: “Empirically, for CIFAR-10, we observed that images with similarity scores above 0.97 were nearly identical, whereas for CelebA, the threshold was approximately 0.95, and for LSUN Church, images with similarity above 0.90 were sometimes, though not always, nearly identical.” The same is repeated in Section 4.1.\n6.\tThe definition of Differential privacy does not come from Dwork & Roth 2014. The first paper that proposed the mathematical framework was [2].\n7.\tWhy is only a single membership inference attack used by Matsumoto et al. 2023a? The latest “evaluation of state-of-the-art MIAs on diffusion models reveals critical flaws and overly optimistic performance estimates in existing MIA evaluation” [3]\n8.\tWhy are only the binary labels used and not more labels in the CPSample?\n9.\tFrom Section 4.1: Figure 1 does not show the most similar pairs of samples and fine-tuning data points.\n\n**References:**\n\n[1] dp-promise: Differentially Private Diffusion Probabilistic Models for Image Synthesis. Haichen Wang, Shuchao Pang, Zhigang Lu, Yihang Rao, Yongbin Zhou, Minhui Xue. USENIS 2024.\n\n[2] Calibrating noise to sensitivity in private data analysis. Dwork, C., McSherry, F., Nissim, K., & Smith, A. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006. \n\n[3] Real-World Benchmarks Make Membership Inference Attacks Fail on Diffusion Models. Chumeng Liang, Jiaxuan You 2024: https://arxiv.org/abs/2410.03640", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The paper tries to solve a very important problem: how to prevent the diffusion models from generating the training samples.", "weaknesses": "1.\tThis method is very simple: “Overfit a classifier on random binary labels assigned to the training data and use this classifier during sampling to guide the images away from the training data.” Does it still require training the classifier on the large training dataset of diffusion models? What if we do not have access to the training data any longer?\n2.\tAs stated in Section 2.1: “Currently, the state-of-the-art for guided generation is achieved by models with classifier-free guidance (Ho & Salimans, 2022). However, since CPSample employs a classifier to prevent replication of its training data,” Thus, the CPSample method is not applicable to the SoTA diffusion models. \n3.\tThe proposed CPSample method is very similar to the AMG guidance method from Chen et al. 2024. It is claimed that AMG requires access to the training data of a diffusion model. How is the classifier trained for the proposed CPSample method without requiring access to the training data? (as stated at the end of Section 3.1)\n4.\tThe qualitative results in Figure 3 clearly show the degradation of quality in the generated images between DDIM and CPSample (especially for the LSUN Church dataset). Additionally, Image C in the last row for the CPSample shows the complete degradation in the quality of the generated images. \n5.\tWhat is the overhead of the method in terms of the inference time? How long does it take to train the classifier? How is the classifier trained? \n6.\tThe paper only compares the FID score of the CPSample to other methods, however, no comparison is presented in terms of the reduction in the generation of the training data points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pwZC8EHFUE", "reviewer": "Reviewer_t8KP", "summary": "Diffusion models exhibit a tendency to memorize training images and reproduce these training images during inference, which can pose significant privacy concerns. To address this issue, this paper introduces a sampling strategy (CPSample) guided by a trained classifier. This classifier evaluates the memorization score of current generated image and rectifies the sampling trajectories to diverge from the training images. By adjusting the guidance scale, the proposed method ensures that the generated images possess a memorization score approximating 0.5, thereby mitigating the risk of replicate generation.", "questions": "1. In my opinion, the trained overfitted classifier is designed to estimate a memorization score and captures a boundary between the member set and hold-out set to provide a guided gradient. However, various alternative methods exist for estimating this boundary or providing a similar memorization gradient. For instance, training a classifier to differentiate between the member set and hold-out set, or training an overfitted diffusion model to provide memorization gradients, are both plausible approaches. Therefore, why choose to assign random 0/1 labels to the member set? (The reasons provided in Line 238-244 is not very convincing for me) \n\n2. The scalability of CPSample to diffusion models with larger member sets is uncertain. Current experiments only provide limited examples (Section 4.2). Additional examples are required to further demonstrate the effectiveness of the proposed method. Furthermore, the hyperparameters ($\\alpha$, $s$) for CPSample seems dependent on each sampling. How to define these hyperparameters for each sampling? \n\n3. The experiments on similarity reduction have been conducted on low-resolution and small-scale datasets such as Cifar10 and CelebA. For these datasets, the diffusion model can verbatim memorize each pixel of the training samples. However, in more practical scenarios, diffusion models tend to memorize only the rough structure of the training sample and can only reproduce similar patterns. Consequently, the current experiments (Section 4.1) may not adequately demonstrate the effectiveness of the proposed method for similarity reduction. Additional experiments on large-scale diffusion models are necessary to further validate the method.\n\n4. The current experiments have been mostly conducted on unconditional diffusion models. To fully evaluate the effectiveness of CPSample, additional experiments on conditional diffusion models, such as text-to-image diffusion models, should be conducted to assess whether CPSample can accurately align with the given condition.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The issue of replicate generation in diffusion models is critical, and there is an urgent need for a scalable solution to address it.\n2. The concept of explicitly training a classifier to measure the memorization score is novel, and subsequent experiments have demonstrated its efficacy.", "weaknesses": "1. The scalability of the strategy to train a classifier is a primary concern. Given that current text-to-image diffusion models, such as Stable Diffusion and Pixart, are trained on large-scale datasets scraped from the Internet, is it a feasible solution to train an overfitted classifier on a large scale dataset? \n\n2. The proposed method assumes that the training member set is available. However, this assumption does not always hold. Specifically, the training member sets of certrain text-to-image diffusion models, such as Pixart, are not publicly accessible, therby limiting the applicability of the proposed approach.\n\n3. While the proposed CPSample method effectively mitigates replicate generation, it also exhibits side effect on the generated images as it manipulates the sampling trajectory. For example, in Figure 4, the generated image by CPSample does not align with the input prompt and sometimes will generate corrupted results. In contrast, although rejection sampling faces challenges with resampling, it tends to produce images that are more closely aligned with the input prompt. \n\n4. There are many typos in notations of DDPM, for example, $\\alpha_t \\rightarrow \\bar{\\alpha}_t$ in Equation 2.1&2.2, Line 146. Please check this through the whole manuscripts to ensure the consistency and accuracy of all notations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IT5zzC4xKQ", "reviewer": "Reviewer_dYdC", "summary": "This paper proposes a rejection sampling method that avoids diffusion models from replicating training data in generated contents. The main idea is to use classifier guidance dependent  on the label of training data / hold-out data. This gets the sampling process away from the region of training data and then preventing diffusion models from generating training data.", "questions": "1) Can you show how your method performs on large-scale datasets?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The method is sound and solid. It implements the idea of rejection sampling by using classifier guidance. This achieves training-free prevention of training data replication.\n\n2) Experiments show that the proposed method can avoid training data replication even if there training setup leads to potential over-fitting. This proves the effectiveness of the proposed method,", "weaknesses": "1) Motivation: the main weakness of this paper lies in its motivation. As shown in existing research, training data replication is not a common phenomena in real-world pre-trained diffusion models. Most training samples are even unable to be inferred as member by membership inference attacks in Stable Diffusion, while the exception ones can be handled case by case. While the proposed method still degrades the generation performance as well as the diversity (potentially), it seems uneconomical to deploy the proposed method on any pre-trained diffusion models. Since privacy issues only happen in pre-trained diffusion models, the significance of the proposed method is moderate.\n\n2) Computation cost & Generalizability: As stated in the paper, we need to train a classifier on all training and some hold-out examples. This could be significantly computationally costly without promising generalizability, since no one has trained such a classifier before. In fact, the training of the classifier in the paper only includes less than 2 thousands images. However, the training datasets of pre-trained diffusion models consist of billions of images. It leaves doubts whether a classifier can effectively give guidance based on these large-scale datasets.\n\n3) OOD generation (minor): One potential drawback, as shown in Figure 4, is that the proposed method may generate images with contents quite different to the description of the text.\n\nIn general, although the proposed method is interesting and performs soundly on toy diffusion models, it does not seem to practical to deploy it on real-world AIGC applications which raises the data privacy concerns. Hence, I do not recommend accepting this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)