CPSample: Classifier Protected Sampling for Guarding Training Data During Diffusion

## Abstract

Abstract Diffusion models have a tendency to exactly replicate their training data, especially when trained on small datasets. Most prior work has sought to mitigate this problem by imposing differential privacy constraints or masking parts of the training data, resulting in a notable substantial decrease in image quality.
We present CPSample, a method that modifies the sampling process to prevent training data replication while preserving image quality.
CPSample utilizes a classifier that is trained to overfit on random binary labels attached to the training data.
CPSample then uses classifier guidance to steer the generation process away from the set of points that can be classified with high certainty, a set that includes the training data.
CPSample achieves FID scores of 4.97 and 2.97 on CIFAR-10 and CelebA-64, respectively, without producing exact replicates of the training data.
Unlike prior methods intended to guard the training images, CPSample only requires training a classifier rather than retraining a diffusion model, which is computationally cheaper.
Moreover, our technique provides diffusion models with greater robustness against membership inference attacks, wherein an adversary attempts to discern which images were in the model’s training dataset.
We show that CPSample behaves like a built-in rejection sampler, and we demonstrate its capabilities to prevent mode collapse in Stable Diffusion.

## 1 Introduction

Diffusion models are an emerging method of image generation that has surpassed GANs on many common benchmarks [dhariwal2021diffusion], achieving state-of-the-art FID scores on CIFAR-10 [cifar], CelebA [celeba], ImageNet [imagenet], and other touchstone datasets.
Although their capabilities are impressive, diffusion models still suffer from the tendency to exactly replicate images found in their training sets [carlini, jagielski2023measuring, somepalli2022diffusion].
This problem is especially pronounced when the training set contains duplicates [laion].
Given that diffusion models are sometimes trained on sensitive content, such as patient data [kazerouni2023diffusion, pinaya2022brain] or copyrighted data [dhariwal2021diffusion], this behavior is generally unacceptable.
Indeed, Google, Midjourney, and Stability AI are already facing lawsuits for using copyrighted data to train image generation models [other_lawsuits, google_lawsuit], some of which exactly replicate their training data during inference [plagiarism].

The strongest formal guarantee against replicating or revealing training data is differential privacy (DP) [dwork].
Unfortunately, differential privacy is at odds with generation quality.
Although differentially private training has been implemented for GANs (DP-GAN) [xie2018differentially], diffusion models (DPDM, DP-Diffusion) [dockhorn2023differentially, better_dpdm], and latent diffusion models (DP-LDM) [latent_dpdm], it typically results in significant degradation of image quality.
Moreover, one cannot easily make a pretrained model differentially private, implying that to achieve differential privacy, one must retrain from scratch.
This makes negotiating the trade-off between privacy and quality challenging, as trying different levels of privacy requires retraining.
Due to the difficulty of achieving differential privacy while simultaneously maintaining quality, some researchers have pursued more attainable model characteristics that have the same flavor as differential privacy.
A frequent benchmark for privacy is robustness to membership inference attacks [hu2023membership], whereby the attacker aims to infer whether a given image was used to train the model.
Although researchers have devised a multitude of loss and likelihood-based membership inference attacks, so far, there are few existing methods that explicitly aim to defend against these attacks besides differential privacy and data augmentation [mem_inf, pang2023whitebox].
A second privacy benchmark measures the cosine similarity in a feature space of a generated image to its nearest neighbor in the training data [daras2023ambient, douze2024faiss].
Ambient diffusion [daras2023ambient] is one method to prevent excessive similarity to the training data without enforcing differential privacy; however, ambient diffusion still has notable negative effects on FID score.

Figure: Figure 1: Generated image and most similar training image pairs for DDIM sampling (left) and CPSample with $\alpha{=}0.001$, $s{=}1000$ (right). We sample $100$ images and display the four with the highest similarity to their nearest neighbors in the training data.
Refer to caption: extracted/5844491/CelebASim.png

Until recently, preventing image replication by diffusion models has involved various forms of data corruption during training, either by adding noise to gradients [dpsgd], diversifying images and captions [somepalli2023understanding], or corrupting the images themselves [daras2023ambient].
Hyperparameter tuning for these methods requires retraining, making it difficult to calibrate them to the necessary level of privacy.
Simple alternatives, like rejection sampling, are effective because they can guarantee that the training images will not be exactly replicated. However, standard rejection sampling has major drawbacks, too. For instance, rejection sampling redistributes probability mass in an inefficient way and requires resampling, which can decrease speed. In extreme cases of mode collapse such as those uncovered by Webster [webster2023reproducible], Stable Diffusion must be queried dozens or even hundreds of times before producing an original image, making rejection sampling impractical.
Rejection sampling is also prone to membership inference attacks and privacy leakages [awan2023privacy].

We present classifier-protected sampling (CPSample), a diffusion-specific data protection technique that, while not strictly differentially private, fortifies against some membership inference attacks and greatly reduces excessive similarity between the training and generated data.
The basic idea is to overfit a classifier on random binary labels assigned to the training data and use this classifier during sampling to guide the images away from the training data.
We show that our method has an effect similar to rejection sampling while removing or reducing the need to resample.
Unlike rejection sampling, CPSample offers protection against some membership inference attacks during the generation process rather than only protecting the end product. CPSample achieves SOTA image quality, improving over previous data protection methods,such as ambient diffusion, DPDMs, and PAC Privacy Preserving Diffusion Models [xu2023pac] for similar levels of “privacy.”
Unlike other methods designed to shield the training data, one can simply adjust the level of protection provided by CPSample without retraining the classifier used for guidance.
CPSample is applicable to existing image models without any expensive retraining of the diffusion models.
We summarize the primary contributions of our work as follows:

- •
In Section 3.1, we introduce CPSample, a novel method of classifier-guidance for privacy protection in diffusion models that can be applied to existing models without retraining.
- •
We show theoretically in Section 3.2 and empirically in Section 4.1 that CPSample prevents training data replication in unguided diffusion.
We also provide evidence in Section 4.2 that CPSample can protect text-based image generation models like Stable Diffusion.
- •
We give empirical evidence that CPSample can foil some membership inference attacks in Section 4.3.
- •
We demonstrate in Section 4.4 that CPSample attains better FID scores than existing methods of privacy protection while still eliminating replication of the training data.

## 2 Background and Related Work

### 2.1 Diffusion Models

We begin with a review of diffusion models.
Denoising diffusion probabilistic models (DDPMs) [sohldickstein2015deep, ho2020denoising] gradually add Gaussian noise to image data during the “forward” process.
Meanwhile, one trains a “denoiser” to predict the original image from the corrupted samples in a so-called “backward” process.
During the forward process, one assigns

$$ $x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon$ (2.1) $$

where $\epsilon\sim\mathcal{N}(0,I)$, $x_{0}$ is the original image, and $\alpha_{t}$ indicates the noise schedule. The variable $t\in\{0,...,T\}$ specifies the step of the forward process, where $x_{0}$ represents an image in the training data.
When $\alpha_{T}$ is set sufficiently close to 0 0, $x_{T}$ is approximately drawn from a standard normal distribution.
During intermediate steps, the distribution of $x_{t}$ is

$$ $q(x_{t}\mid x_{0})=\mathcal{N}(x_{t};\sqrt{\alpha_{t}}x_{0},(1-\alpha_{t})I).$ (2.2) $$

During training, one performs gradient descent on $\theta$ to minimize the score-matching loss, given by

$$ $\mathbb{E}_{\epsilon\sim\mathcal{N}(0,1),x_{0}\sim\mathcal{D}}\left[\sum_{t=1} ^{T}\frac{1}{2\sigma_{t}^{2}}\|\epsilon-\epsilon_{\theta}(\sqrt{\overline{ \alpha}_{t}}x_{0}+\sqrt{1-\overline{\alpha}_{t}}\epsilon,t)\|^{2}\right].$ (2.3) $$

Here, $\mathcal{D}$ is the target distribution, which is approximated by sampling from the training data.
Finally, to generate a new image, one samples standard Gaussian noise $x_{T}\sim\mathcal{N}(0,I)$. Then, one gradually denoises $x_{T}$ by letting

$$ $x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1- \overline{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t)\right)+\sigma_{t}z_{t}$ (2.4) $$

In each step, $z_{t}\sim\mathcal{N}(0,I)$, and $\sigma_{t}$ and $\alpha_{t}$ are scalar functions determined by the noise schedule that govern the rate of the backwards diffusion process.

Despite the superior image quality afforded by DDPM, the sampling process sometimes involves $1\,000$ or more steps, which led to a variety of sampling schemes and distillation methods for speeding up inference [song2022denoising, kim2023consistency, song2023consistency, gu2023boot].
One of the most commonly used modifications to the sampling process is denoising diffusion implicit models (DDIM), which enables skipping steps in the backward process.

Currently, the state-of-the-art for guided generation is achieved by models with classifier-free guidance [ho2022classifierfree].
However, since CPSample employs a classifier to prevent replication of its training data, it is more useful for us to review its predecessor, classifier-guided diffusion [score_based, dhariwal2021diffusion]. In classifier guided diffusion, a pretrained classifier $p_{\phi}(y\mid x_{t},t)$ assigns a probability to the event that $x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon$ for some $x_{0}$ with label $y$. The sampling process for
classifier-guided DDIM is modified by

$$ $\displaystyle\hat{\epsilon}_{t}=\epsilon_{\theta}(x_{t})-\sqrt{1-\overline{ \alpha}_{t}}\nabla_{x_{t}}\log p_{\phi}(y\mid x_{t},t)$ (2.5) $\displaystyle x_{t-1}=\sqrt{\overline{\alpha}_{t-1}}\left(\frac{x_{t}-\sqrt{1- \overline{\alpha}_{t}}\hat{\epsilon}_{t}}{\sqrt{\overline{\alpha}_{t}}}\right) +\sqrt{1-\overline{\alpha}_{t}}\hat{\epsilon}_{t}.$ (2.6) $$

Such a modification of the sampling procedure corresponds to drawing $x_{t}$ from the joint distribution:

$$ $\displaystyle p_{\theta,\phi}(x_{t},y\mid x_{t+1},t)=Zp_{\theta}(x_{t}\mid x_{ t+1},t)p_{\phi}(y\mid x_{t},t)$ (2.7) $$

where $Z$ is a normalization constant. This formulation can be adapted for continuous-time models, but for discrete-time models, additional care must be taken to ensure accuracy (see Appendix A for additional details).

### 2.2 Privacy in Diffusion Models

Differential privacy (DP) is generally considered to be the gold standard for protecting sensitive data.
The formal definition of ($\epsilon$-$\delta$) differential privacy is as follows [dwork]:

###### Definition 2.1 (( $\epsilon$ - $\delta$ )-Differential privacy) .

Let $\mathcal{A}$ be a randomized algorithm that takes a dataset as input and has its output in $\mathcal{X}$. If $D_{1}$ and $D_{2}$ are data sets with symmetric difference $1$, then $\mathcal{A}$ is $\epsilon$-$\delta$ differentially private if for all $S\subset\mathcal{X}$,

$$ $\displaystyle\mathbb{P}(\mathcal{A}(D_{1})\in S)\leq\mathbb{P}(\mathcal{A}(D_{ 2})\in S)e^{\epsilon}+\delta$ (2.8) $$

DP ensures that the removal or addition of a single data point to the dataset does not significantly affect the outcome of the algorithm, thus protecting the identity of individuals within the dataset.
Existing DP diffusion models [dockhorn2023differentially, better_dpdm, latent_dpdm] achieve DP through DP stochastic gradient descent (DP-SGD) [dpsgd], in which Gaussian noise is added to the gradients during training.

Though DP offers a formal guarantee that one’s data is secure, imposing a DP constraint in practice severely compromises the quality of the synthetic images. Therefore, researchers have largely resorted to demonstrating that models exhibit various relaxations of strict DP, such as Probably Approximately Correct DP (PAC-DP) [xiao2023pac] or other empirical metrics of privacy.
For instance, researchers measure the distance of generated images to their nearest neighbors in the training set and try to ensure that the number with similarity exceeding some threshold is small [daras2022soft, daras2023ambient].
Usually, one computes similarity either via least squares or cosine similarity, given by

$$ $\displaystyle\frac{x^{T}\cdot\mathrm{C}(x)}{\lVert x\rVert\cdot\lVert\mathrm{C }(x)\rVert},$ (2.9) $$

with $\mathrm{C}(x)$ denoting the nearest neighbor of $x$ among the training data.
Usually the cosine similarity is computed in a feature space rather than the raw pixel space [face_verification, XIA201539]. For CIFAR-10, we observed that images with similarity scores above $0.97$ were nearly identical, whereas for CelebA, the threshold was approximately $0.95$.
For LSUN Church, images with similarity above $0.90$ were sometimes, though not always, nearly identical.
While preventing exact replication of training data does not imply DP, a DP model will not exactly reveal members of its training data with high probability.
Therefore, preventing exact replication of training data recovers one of the desirable implications of DP.

Ambient diffusion reduces similarity to the nearest neighbor by masking pixels during training and only scoring the model based on the visible pixels.
In this way, the model never has access to a full, uncorrupted image and is less likely to replicate full images, as it has not seen them [daras2023ambient].
The downside is that masking pixels, even at the relatively modest rate of $20\%$, leads to notable image quality degradation, measured via a notable increase in the FID score.
Moreover, ambient diffusion shifts the entire distribution of similarity scores towards lower similarity, whereas we should ideally prevent the generation of images with high similarity to the training data while leaving the rest of the distribution untouched. Tools have been developed to help compute nearest neighbors efficiently, since naïve pairwise comparisons are too computationally expensive.
In 2023, MetaAI developed the FAISS library for efficient similarity search using neural networks [douze2024faiss], making this type of privacy metric possible to compute approximately in a reasonable amount of time.

Until recently, all attempts at enforcing privacy for diffusion models occurred during training.
In 2023, Xu et al. [xu2023pac] developed a method of classifier-guided sampling (PACPD) that has PAC privacy advantages over standard denoising.
For text-guided models, Somepalli et al. [somepalli2023understanding] developed a method of randomly changing influential input tokens to avoid exact memorization and Wen et al. [wen2024detecting] protected training data using a regularization technique on the classifier-free guidance network during training.

###### Definition 2.1 (( ϵ italic-ϵ \epsilon italic_ϵ - δ 𝛿 \delta italic_δ )-Differential privacy) .

### 2.3 Membership Inference Attacks

A third privacy measurement comes from membership inference attacks [Dubinski_2024_WACV, pang2023blackbox, duan2023diffusion, wu2022membership], whereby one tries to discern whether a given data point was a member of the training set for the model. Robustness to membership inference attacks is implied by differential privacy. Membership inference attacks against diffusion models usually hinge on observed differences in reconstruction loss or likelihood that come from overfitting. In this paper, we will use a slight modification of the membership inference attack from [matsumoto2023membership] given in Algorithm 1 described in Appendix E.

We repeat Algorithm 1 for $x_{1},...,x_{m}$ drawn first from the training data, and then from withheld test data. If the resulting mean reconstruction error is significantly higher for test data than for training data, then we say that the diffusion model has failed the inference attack.

## 3 Protecting Privacy During Sampling

In this section, we address the problem of training data replication in diffusion models, which poses significant privacy risks. One common solution to this problem is rejection sampling, where samples that closely resemble training data are discarded. However, rejection sampling has several shortcomings: it is computationally expensive, inefficient, and does not provide protection during the sampling process itself, but only in the final output. Moreover, in extreme cases of mode collapse, one must generate dozens of images before generating original content when rejection sampling.

To overcome these limitations, we introduce CPSample, a method that reproduces some of the benefits of rejection sampling without the need for resampling. CPSample integrates classifier guidance into the sampling process to steer the generation away from the training data. By overfitting a classifier on random binary labels assigned to the training data, we use this classifier during sampling to adjust the generated images, thus reducing the likelihood of replicating training data while maintaining image quality.

Let $B_{\delta}(x)$ denote the ball of radius $\delta$ around $x$ in a given metric space. Explicitly, for a training dataset $D$, $\delta>0$ that is not too large, and $\epsilon>0$, our goal is to produce a diffusion model that generates data $\tilde{x}$ such that $\mathbb{P}\left(\tilde{x}\in\bigcup_{x\in D}B_{\delta}(x)\right)<\epsilon$ while compromising image quality as little as possible in the process.

### 3.1 Sampling Method

The first step in CPSample is to train a network that can provide information about how likely a sample $x_{t}$ is to turn into a member of the training data at the end of the denoising process. For this task, we use a classifier trained to memorize random binary labels assigned to the training data, which is possible for a sufficiently over-parameterized network relative to the number of points being memorized [random_labels_orig]. During the denoising process, whenever the classifier predicts a label $y\in\{0,1\}$ for $x_{t}$ with probability greater than $1-\alpha$, we perturb $x_{t-1}$ towards the opposite label using classifier guidance. Without loss of generality, if the classifier predicts the label $1$ with high probability, we employ classifier guidance to adjust the sampling process. Specifically, we aim to sample from a modified distribution that reduces the likelihood of the generated sample being close to the training data. We employ classifier guidance to perturb the denoising process to sample from the conditional distribution $p_{\theta,\phi}(x_{t-1}\mid x_{t},t,y=0)$. This approach ensures that the modified sampling process leads to a lower density near points in the training data that $x_{t}$ might have otherwise converged to.

To state our procedure more precisely, let $\epsilon_{\theta}(\cdot,\cdot)$ be the denoiser. Note that the classifier is trained once on the training data and not during each sample generation. The sampling process is then modified in the following steps:

- 1.
Randomly assign Bernoulli$(0.5)$ labels to each member of the training data, and let $B\in\{0,1\}^{n}$ index these random labels. Train a classifier $p_{\phi}(y\mid x_{t},t)$ to predict these labels. Here, $x_{t}$ is generated by corrupting the training data $x_{0}$ with noise: $x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon$ for $\epsilon\sim\mathcal{N}(0,I)$ and $t\in\{0,...,T\}$.
- 2.
Set a tolerance threshold $0<\alpha<0.5$ and a scale parameter $s$. Let $p_{\phi}(y\mid x_{t},t)$ be the probability assigned to the label $y$ by the classifier $p_{\phi}(y\mid x_{t},t)$. Sample $x_{T}\sim\mathcal{N}(0,I)$. For $t\in\{T,....,1\}$, if $p_{\phi}(y=0\mid x_{t},t)<\alpha$, replace $\epsilon_{\theta}(x_{t},t)$ with
$\hat{\epsilon}_{\theta,\phi}(x_{t},t)=\epsilon_{\theta}(x_{t},t)-s\sqrt{1-
\overline{\alpha}_{t}}\cdot\nabla\log(p_{\phi}(y=0\mid x_{t},t)).$
If $p_{\phi}(y=1\mid x_{t},t)<\alpha$, replace $\epsilon_{\theta}$ with
$\hat{\epsilon}_{\theta,\phi}(x_{t},t)=\epsilon_{\theta}(x_{t},t)-s\sqrt{1-
\overline{\alpha}_{t}}\cdot\nabla\log(p_{\phi}(y=1\mid x_{t},t)).$
Otherwise, we leave the sampling process unchanged.

Though the choice of random labels for the classifier initially may seem counter-intuitive, it has several advantages over other approaches. If we used labels corresponding to real attributes of the data, the classifier would influence the content of the generated images in ways that could compromise their authenticity and diversity. This is because the guidance would push the generated images towards or away from specific attributes, altering the intended distribution. The perturbation applied by the gradient of the log probability in CPSample moves the generated images away from regions where they can be easily classified as similar to the training data. This method is more effective than adding random noise, which would require a significant amount of noise to achieve the same effect, thus degrading image quality.

Moreover, without a classifier, detecting when the denoising process is likely to recreate a training data point would be challenging. The classifier provides a mechanism to identify and steer image generation away from such points. If we trained the classifier to distinguish between train and test data and guided the diffusion process towards the test data, we might inadvertently replicate or reveal aspects of the test data.

Unlike past training-based methods of privacy protection, once we have trained the classifier, we can adjust the level of protection by tuning the hyperparameters $s$ and $\alpha$ without necessitating retraining of the classifier or denoiser. In ambient diffusion and DPDM, if one realizes after training that one needs a higher or lower level of privacy, one must retrain the entire diffusion model from scratch.

### 3.2 Theory

In this section, we show that CPSample functions similarly to rejection sampling when preventing exact replication of the training images. We work under the following assumptions:

###### Assumption 1 .

Suppose that the classifier $p_{\phi}(y\mid x,t)$ has Lipschitz constant $L$ in the argument $x$ with respect to a metric $d(\cdot,\cdot):\chi\times\chi\rightarrow\mathbb{R}_{\geq 0}$, where $\chi$ denotes the image space.

###### Assumption 2 .

Let $y_{i}$ be the random label assigned to $x_{i}\in D$, where $D$ is the training data.
Suppose that the classifier $p_{\phi}$ is trained sufficiently well such that for $\gamma<1$,

$$ $\sum_{x_{i}\in D}\mathbb{P}\left(p_{\phi}(y_{i}\mid\sqrt{\alpha_{t}}x_{i}+ \sqrt{1-\alpha_{t}}\epsilon,t)\notin(1-\kappa,1]\right)<\gamma$ (3.1) $$

for all $x_{i}\in D$, $t\in[0,T]$ and $\epsilon\sim\mathcal{N}(0,I)$.

###### Assumption 3 .

Suppose that CPSample generates data $\tilde{x}$ such that $\lambda<p_{\phi}(y\mid\tilde{x},0)<1-\lambda$ with probability greater than $1-\nu$, where we are able to govern $\nu$ and $\lambda$ by adjusting $s$ and $\alpha$ in Section 3.1.

In Assumption 1 the constant $L$ can be difficult to evaluate, but the assumption holds for neural network classifiers. Methods exist that can bound the local Lipschitz constant around the training data [local_lipschitz], which one can use to strengthen the guarantees of Lemma 1.
Assumptions 2 and 3 hold well empirically.
We were able to train our classifier to have a cross-entropy loss below $0.05$ in the experiments from Sections 4.1 and 4.2. Moreover, during sampling, we observed that CPSample had control over the quantity $p_{\phi}(y\mid x_{t},t)$. An example is given in Figure 6.

Given these assumptions, we can demonstrate the following simple lemma, which links the behavior of CPSample to that of a rejection sampler without requiring expensive comparisons to the training data set. A proof can be found in Appendix A.

###### Lemma 1 .

Under the above assumptions, choose $\epsilon>0$ and $0<\delta<\frac{\frac{1}{2}-\kappa}{L}$. Setting $\nu=\epsilon$ and $\lambda=\kappa+L\delta$, when drawing a single sample, with probability greater than $(1-\epsilon)(1-\gamma)$, CPSample generates an image that lies outside of $S=\bigcup_{x\in D}B_{\delta}(x)$ in the metric space defined by $d$.

Note that the ability to control $\mathbb{P}\left(\tilde{x}\in\bigcup_{x\in D}B_{\delta}(x)\right)$ gives the same guarantee offered by rejection sampling. However, in extreme instances of mode collapse such as those exhibited by Stable Diffusion in Section 4.2, one might have to resample hundreds of times to generate original images, making standard rejection sampling highly inefficient. CPSample is able to produce original images without this high level of inefficiency.

###### Assumption 1 .

###### Assumption 2 .

###### Assumption 3 .

###### Lemma 1 .

## 4 Empirical Results

Figure: Figure 2: Generated image and most similar training image pairs for DDIM sampling and CPSample with $\alpha=0.001,s=1$ on CIFAR-10 (left) and $\alpha=0.1,s=10$ on LSUN Church (right). For each pair, the image on the left is the generated sample and the one on the right is its nearest neighbor in the training set. These are the four examples out of $21\,000$ images on CIFAR-10 and two out of $1\,700$ images on LSUN Church that have the highest similarity scores with their nearest neighbor.
Refer to caption: x1.png

We run three distinct sets of experiments to demonstrate the ways in which CPSample protects the privacy of the training data. First, we statistically test the ability of CPSample to reduce similarity between generated data and the training set for unguided diffusion. Then, we demonstrate that CPSample can prevent Stable Diffusion from generating memorized images. Finally, we measure robustness against membership inference attacks. Hyperparameters in all empirical tests were chosen to maximize image quality while eliminating exact matches.

Figure: Figure 3: Cosine similarity in feature space between generated images and their nearest neighbor in the fine-tuning data set for standard DDIM sampling (red) and CPSample (blue) on CIFAR-10 ($\alpha=0.001,s=1$) and CelebA-64 $(\alpha=0.001,s=1000)$. Similarity scores were computed for $21\,000$ generated samples for CIFAR-10 and $8\,000$ images for CelebA. Note that standard DDIM exhibits many more samples with similarity scores exceeding the thresholds from Table 1.
Refer to caption: x2.png

**Table 1: Reduction in cosine similarity between generated images and nearest neighbor in fine-tuning data.**
| Dataset | FT Steps | $\alpha$ | Scale | Threshold | DDIM | CPSample | p-value^1 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10 | 150k | 0.001 | 1 | 0.97 | 6.25$\%$ | 0.00 $\%$ | <0.0001 |
| CelebA | 650k | 0.001 | 1000 | 0.95 | 12.5$\%$ | 0.10$\%$ | <0.0001 |
| LSUN Church | 455k | 0.1 | 10 | 0.90 | 0.73$\%$ | 0.04$\%$ | 0.013 |

### 4.1 Similarity Reduction

We generate images using DDIM with CPSample and $1000$ denoising steps. The nearest neighbor to each generated image was found using Meta’s FAISS model [douze2024faiss]. Similarity between two images is measured by cosine similarity in a feature space defined by FAISS. A similarity score exceeding 0.97 often indicates nearly-identical images for CIFAR-10. For CelebA and LSUN Church, the thresholds lie around 0.95 [daras2023ambient] and $0.90$ respectively. Note that a cosine similarity score above the thresholds given is a necessary but not sufficient condition for images to look very alike. To ensure that we could observe a larger number of images with similarities exceeding our thresholds, we fine-tuned the models using DDIM [song2022denoising]
on a subset of the data that consisted of 1000 images, as was done in [daras2023ambient]. This modification allows us to statistically test the efficacy of CPSample without the large number of samples required to do hypothesis testing on rare exact replication events.
After fine-tuning, up to $12.5\%$ of the images produced by unprotected DDIM were nearly exact replicates of the fine-tuning data. One can see from Table 1 that CPSample significantly reduces the fraction of generated images that have high cosine similarity to members of the fine-tuning set. One can see histograms of the similarity score distribution with and without CPSample in Figures 3 and 9. Figures 1 and 2 show the most similar pairs of samples and fine-tuning data points. Uncurated images generated from CPSample can be found in Appendix F.
While CPSample effectively reduces the similarity between generated images and the training data, our experiments indicate that CPSample achieves minimal degradation in quality compared to previous methods 4.

Figure: Figure 4: The generated and real images with the highest similarity for CIFAR-10 (left) and CelebA (right) out of $50\,000$ samples used to compute FID score.
Refer to caption: extracted/5844491/fid_score_images_aligned.png

### 4.2 Stable Diffusion

As a second demonstration of CPSample, we present evidence that CPSample can prevent well-known examples of mode collapse in near-verbatim attacks against Stable Diffusion [webster2023reproducible, wen2024detecting]. We create a small dataset of commonly reproduced images [somepalli2023understanding] and include other images from the LAION dataset depicting the same subjects. We ensure that this dataset contains no duplicates. In this more targeted application, CPSample can prevent exact replication when used with the right hyperparameters. See Figure 5 and Table 2 for more details. Though CPSample does not provide as robust protection in this setting as in [somepalli2023understanding, wen2024detecting], these results still show the potential of CPSample for data protection in text-guided diffusion models. The methods developed in [somepalli2023understanding, wen2024detecting] do not apply for unguided diffusion models.

**Table 2: Details of generation on Stable Diffusion.**
| Image | Original caption | Modified caption | $\alpha$ | scale | guidance |
| --- | --- | --- | --- | --- | --- |
| A | “Rambo 5 and Rocky Spin-Off - Sylvester Stallone gibt Updates" | “Rocky and Rambo Spin-Off - Sylvester Stallone gibt Updates" | 0.5 | 2000 | 1.5 |
| B | “Classic cars for sale" | “Classic car for sale" | 0.3 | 100 | 1.5 |
| C | “Red Exotic Fractal Pattern Abstract Art On Canvas-7 Panels" | “Red Exotic Fractal Pattern Abstract Art On Canvas-7 Panels" | 0.5 | 2000 | 1.5 |

### 4.3 Membership Inference Attacks

We also assess CPSample ability to protect against membership inference attacks. Following Algorithm 1, we compute the mean reconstruction error for the train and test datasets and determine whether there is a statistically significant difference.
To evaluate resistance to inference attacks, we use a model trained on all $50\,000$ CIFAR-10 training images. We compare reconstruction loss on these $50\,000$ training images to reconstruction loss on the $10\,000$ withheld test samples included in the CIFAR-10 data set. We juxtapose the difference in reconstruction loss between these two datasets for both CPSample, with a classifier $c_{\phi}$ trained on all of the CIFAR-10 training data with random labels, and standard DDIM sampling. We demonstrate resistance to inference attacks for $\alpha\in\{0.5,0.25,0.001\}$ over about $8000$ images from each of the train and test sets. The $p$-values in this experiment are based on a two-sample, single-tailed $Z$-score that tests the null hypothesis “the average training reconstruction loss is less than or equal to the average test reconstruction loss." Let $n$ be the number of training data points and $m$ be the number of test data points sampled. The test statistic is

$$ $\frac{\mu_{\textrm{test}}-\mu_{\textrm{train}}}{\sqrt{V_{\textrm{test}}/m+V_{ \textrm{train}}/n}}.$ $$

The variable $V$ indicates the variance and $\mu$ indicates the mean.
In our case, failure to reject the null indicates success for CPSample.

We observe that in our experiments, a very low value of $\alpha$ leads to a higher $p$-value, which is counter intuitive. We suspect that this occurs because when $\alpha$ is small, it results in a more targeted application of CPSample, driving the loss up exclusively around the training data points. However, for all values of $\alpha$ between 0 0 and $0.5$, one is not able to conduct a conclusive membership inference attack against CPSample. We provide a second black-box membership inference attack based on permutation testing in Appendix E.

### 4.4 Quality Comparison

As mentioned, other methods of privacy protection suffer from severe degradation of quality, as measured by FID score. Here, we provide an FID score comparison between CPSample model finetuned on subsets of CIFAR-10 and CelebA, and existing methods of privacy protection. We exhibit FID scores for unconditional generation of CIFAR-10 and CelebA in Table 4. The images with the highest similarity to the training set, as determined by FAISS, can be found in Figure 5.
We chose $\alpha$ and $s$ by finding the least aggressive settings that completely prevent exact replication of the training data. For more extensive results on how $\alpha$ and $s$ affect FID score, see Table 7.

**Table 3: Difference in mean reconstruction error between train and test data for CIFAR-10.**
| Method | Test statistic | p-value |
| --- | --- | --- |
| DDIM | 138 | $\approx$ 0 |
| Ambient (Corruption $0.2$) | 0.141 | 0.44 |
| Ambient (Corruption 0.8) | -0.024 | 0.51 |
| CPSample ($\alpha=0.5$) | 0.59 | 0.28 |
| CPSample ($\alpha=0.25$) | 0.23 | 0.41 |
| CPSample ($\alpha=0.001$) | -0.86 | 0.81 |

## 5 Limitations

While CPSample prevents training data replication well, it comes with the drawback that it can be difficult to train a classifier on binary random labels for large data sets, making CPSample better suited to protecting small to moderately sized datasets. Nevertheless, we were able to successfully protect datasets of up to $180\,000$ images. With larger datasets, it could become difficult to obtain a classifier that has high accuracy while maintaining a reasonably low Lipschitz constant without introducing Lipschitzness regularizations. When training only on a few images, we noticed instability in the classifier for Stable Diffusion that made it difficult to guarantee complete protection across seeds and at higher levels of classifier-free guidance, which could be attributed to non-pretrained classifier models. Additional research on regularization could help to alleviate this limitation.

If there are multiple exact repeats in the training set, CPSample could struggles to protect the training data if opposite labels were assigned to the repetitions, though it still can perform well when there are multiple near-repeats. Research surrounding removing repeats or randomly transforming the data to prevent exact repeats would likely improve CPSample’s performance. Repeats could also be strategically grouped to have the same label.

Finally, given the difficulty of measuring the Lipschitz constant of a neural network, it may be difficult to give practical bounds on the protection rate of CPSample.
In practice, we observe stronger protections than the formal guarantees provide.

## 6 Conclusion

We have presented a new approach to prevent memorized images from appearing during inference time. Our method is applicable to both guided and unguided diffusion models. Unlike previous methods intended to protect privacy of unguided diffusion models, CPSample does not necessitate retraining the denoiser. Moreover, after training the classifier, one can adjust the level of protection enforced by CPSample without further training. We have shown theoretically that our method behaves similarly to rejection sampling without necessitating resampling. Finally, we have provided empirical evidence with rigorous statistical testing that our method is effective in unguided settings. We have also given examples in which CPSample was able to prevent extreme instances of mode collapse in Stable Diffusion. Despite its efficacy at preventing replication of training images, CPSample has little negative impact on image quality.

### Broader Impact

Both guided and unguided diffusion models are prone to reproducing their training data and consequently violating copyright or privacy.
As diffusion models become more widespread in entertainment and commercial settings, these issues have the potential to cause real harm. We explore a new avenue to prevent replication and promote robustness to membership inference attacks through early detection and perturbation. When deployed responsibly, CPSample has the potential to prevent the majority of instances in which exact replication occurs.

## Appendix A Proofs

#### Details of classifier guidance

For completeness, we include a derivation of the classifier-guidance introduced in [dhariwal2021diffusion].

During the conditional denoising process, one should sample $x_{t-1}$ from the conditional distribution

$$ $\mathbb{P}(x_{t-1}\mid x_{t},y)=\frac{\mathbb{P}(x_{t-1},x_{t},y)}{\mathbb{P}( x_{t},y)}=\frac{\mathbb{P}(x_{t-1}\mid x_{t})\mathbb{P}(y\mid x_{t},x_{t-1})}{ \mathbb{P}(x_{t},y)}.$ (A.1) $$

One can show that $\mathbb{P}(y\mid x_{t},x_{t-1})=\mathbb{P}(y\mid x_{t-1})$ (see [dhariwal2021diffusion] for details). The denominator $\mathbb{P}(x_{t},y)$ is intractable and does not depend on $x_{t-1}$. Therefore, we write this term as $Z$. To get an estimate of the probability $\mathbb{P}(y\mid x_{t-1})$, we train a classifier of the form $p_{\phi}(y\mid x_{t-1})$. Thus, we should estimate the conditional probability $\mathbb{P}(x_{t-1}\mid x_{t},y)$ via

$$ $p_{\theta,\phi}(x_{t-1},x_{t},y)=Zp_{\theta}(x_{t-1}\mid x_{t})p_{\phi}(y\mid x _{t-1}).$ (A.2) $$

In continuous time, we can write $p(x_{t},y)=p(x_{t})p(y\mid x_{t})$, and the score function is:

$$ $\displaystyle\nabla_{x_{t}}\log(p_{\theta}(x_{t})p_{\phi}(y\mid x_{t}))$ $\displaystyle=\nabla_{x_{t}}\log p_{\theta}(x_{t})+\nabla_{x_{t}}\log p_{\phi} (y\mid x_{t}).$ (A.3) $$

The network $\epsilon_{\theta}(x_{t},t)$ predicts the noise added to a sample, which can be used to derive the score function

$$ $\nabla_{x_{t}}\log p_{\theta}(x_{t},t)=-\frac{1}{\sqrt{1-\overline{\alpha}_{t} }}\epsilon_{\theta}(x_{t},t).$ $$

Substituting this into (A.3), we get

$$ $-\frac{1}{\sqrt{1-\overline{\alpha}_{t}}}\epsilon_{\theta}(x_{t})+\nabla_{x_{t }}\log p_{\phi}(y\mid x_{t}).$ (A.4) $$

This leads to a new prediction for

$$ $\hat{\epsilon}_{\theta}(x_{t})=\epsilon_{\theta}(x_{t})-\sqrt{1-\overline{ \alpha}_{t}}\nabla_{x_{t}}\log p_{\phi}(y\mid x_{t}).$ $$

The conditional sampling then follows in the same manner as standard DDIM with $\epsilon_{\theta}$ replaced by $\hat{\epsilon}_{\theta}$.

#### Proof of Lemma 1

###### Proof.

Let $x^{\prime}\in B_{\delta}(x_{0})$, where $x_{0}\in D$ is assigned the random label $y$. By the definition of Lipschitz, we have that

$$ $|p_{\phi}(y\mid x_{0},t)-p_{\phi}(y\mid x^{\prime},t)|<Ld(x_{0},x^{\prime}).$ $$

Since $p_{\phi}(y\mid x_{0},0)>1-\kappa$, it follows that

$$ $\displaystyle p_{\phi}(y\mid x^{\prime},0)$ $\displaystyle=p_{\phi}(y\mid x_{0},0)-p_{\phi}(y\mid x_{0},0)+p_{\phi}(y\mid x ^{\prime},0)$ $\displaystyle=p_{\phi}(y\mid x_{0},0)-(p_{\phi}(y\mid x_{0},0)-p_{\phi}(y\mid x ^{\prime},0))$ $\displaystyle\geq p_{\phi}(y\mid x_{0},0)-|p_{\phi}(y\mid x_{0},0)-p_{\phi}(y \mid x^{\prime},0)|$ $\displaystyle\geq p_{\phi}(y\mid x_{0},0)-Ld(x_{0},x^{\prime})$ $\displaystyle\geq p_{\phi}(y\mid x_{0},0)-L\delta$ $\displaystyle\geq 1-\kappa-L\delta$ $\displaystyle=1-\lambda.$ $$

By assumption, CPSample generates samples $\tilde{x}$ with $\lambda<p_{\phi}(y\mid\tilde{x})<1-\lambda$ with probability at least $1-\nu$. Because all points $x^{\prime}\in S$ have $p_{\phi}(y\mid x^{\prime},0)\in[0,\lambda]\bigcup[1-\lambda,1]$ with probability at least $1-\gamma$, CPSample must generate its samples $\tilde{x}\in\chi\setminus S$ with probability at least $(1-\epsilon)(1-\gamma)$.
∎

Figure: Figure 6: CPSample is able to generate images with $p_{\phi}(y\mid\tilde{x},0)\in(\lambda,1-\lambda)$. This example shows the probability $p_{\phi}(y=1\mid x_{t},t)$ during the generation process with Stable Diffusion guided by the caption “Rambo 5 and Rocky Spin-Off - Sylvester Stallone gibt Updates." Note that a higher step indicates a later point in the denoising process. In this example, Stable diffusion exactly replicated the memorized image of Stallone, whereas CPSample $(\alpha=0.5,s=2000)$ produced an original image.
Refer to caption: extracted/5844491/prob_control.png

###### Proof.

## Appendix B Class Guided Diffusion

As a final experiment, we implement CPSample alongside classifier-free guidance for CIFAR-10 to ensure that CPSample does not cause frequent out-of-category samples. The models used for guided diffusion were smaller, so the image quality is naturally lower.

Figure: Figure 7: Uncurated samples using classifier-free guidance on CIFAR-10. The image in the position second row, third column from the top left is a near-exact replica of a member of the training data.
Refer to caption: extracted/5844491/SampledGuidenceImgs.png

Figure: Figure 8: Uncurated samples using CPSample $(\epsilon=0.1,s=10)$ along with classifier-free guidance on CIFAR-10. Note that although CPSample slightly reduces image quality, it does not cause out-of-category samples.
Refer to caption: extracted/5844491/guided_0.1_10.png

## Appendix C Training details

#### Training classifiers.

For training the classifier, we randomly selected subsets of 1000 images each from the CIFAR-10, CelebA, and LSUN Church datasets, on which we trained the classifier from scratch. The architecture of our classifier is a modified version of the U-Net model. We retained key components of the U-Net [ronneberger2015unet] model structure, including the timestep embedding, multiple convolutional layers for downsampling, and middle blocks. The output from the middle blocks underwent processing through Group Normalization, SiLU [elfwing2017sigmoidweighted] activation layers, and pooling layers before being fed into a single convolutional layer, yielding the classifier’s output. Parameters for layers identical to the standard U-Net were consistent with those used to pretrain the DDIM model on these datasets. Additionally, akin to the pretraining of DDIM, we incorporated Exponential Moving Average during training to stabilize the training process. The training of each classifier model was conducted using 4 NVIDIA A4000 GPUs with 16GB of memory. For subsets of $1000$ images, the classifier took only hours to train. For larger datasets consisting of $60,0000-160,000$ data points, the classifier took up to $1$ week to train. By comparison, retraining a diffusion model to be differentially private or using the method presented in [daras2023ambient] can take weeks or months depending on the data set.

#### Fine-tuning pretrained denoiser model on subsets.

For fine-tuning the pretrained denoiser model on subsets, we commenced with the 500,000-step pretrained checkpoints available for the denoiser DDIM model. Fine-tuning was performed on subsets of 1000 images each from the CIFAR-10, CelebA, and LSUN Church datasets until the model began generating data highly resembling the respective subsets. The number of training steps varied across different models, and specific details regarding the fine-tuning process can be found in Table 5. Throughout the fine-tuning process, hyperparameters remained consistent with those used during the pretraining phase. We employed 2 NVIDIA A5000 GPUs with 24GB of memory for fine-tuning each model on the subsets.

**Table 5: Training Parameters & Steps**
|  | Batch Size | LR | Optimizer | EMA Rate | Classifier Steps | Fine-tune Steps |
| --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10 | 256 | 2e-4 | Adam | 0.9999 | 560,000 | 110,000 |
| CelebA | 128 | 2e-4 | Adam | 0.9999 | 610,000 | 150,000 |
| LSUN Church | 8 | 2e-5 | Adam | 0.999 | 1250,000 | 880,000 |

## Appendix D Evaluation Details

#### Numerical stability

For the purposes of numerical stability, we slightly modified the sampling process described in Section 3.1. We noticed in earlier iterations of our method that very small numbers of images were becoming discolored or black because in float16, the classifier was predicting probabilities of 0.0000 or 1.0000 for the random label 1, causing the logarithm to blow up. To fix this in practice, we do the following. Sample $x_{T}\sim\mathcal{N}(0,I)$. For $t\in\{T,....,1\}$, if $p_{\phi}(y=0|x_{t},t)<\alpha$, replace $\epsilon_{\theta}(x_{t},t)$ with

$$ $\hat{\epsilon}_{\theta,\phi}(x_{t},t)=\epsilon_{\theta}(x_{t},t)-s\sqrt{1- \overline{\alpha}_{t}}\cdot\nabla\log(\tau+p_{\phi}(y=0|x_{t},t)).$ $$

If $p_{\phi}(y=1|x_{t},t)<\alpha$, replace $\epsilon_{\theta}$ with

$$ $\hat{\epsilon}_{\theta,\phi}(x_{t},t)=\epsilon_{\theta}(x_{t},t)-s\sqrt{1- \overline{\alpha}_{t}}\cdot\nabla\log(\tau+p_{\phi}(y=1|x_{t},t)).$ $$

Otherwise, leave the sampling process unchanged.

By setting $\tau$ equal to 0.001, we were able to prevent the undesirable behavior.

#### Similarity Reduction Evaluation.

We employ the fine-tuned denoiser model to generate 3000 image samples for each of the aforementioned datasets. Additionally, we utilize the Classifier-guided method to generate another set of 3000 images. Subsequently, we employ DINO [caron2021emerging] to find nearest neighbors in the subset using a methodology akin to ambient diffusion. From the perspectives of both DINO’s similarity scores and human evaluation, we observe that images generated through the classifier-guided approach exhibit significantly lower similarity to the original images in the subset compared to those generated without guidance.

#### FID Evaluation.

For each dataset, we utilize the denoiser model fine-tuned on the subset to generate 30,000 images under the guidance of the classifier. Subsequently, we employ the FID score implementation from the EDM [karras2022elucidating] paper to compute the FID score.

#### Inference Speed

Although speed was not a goal of our method, we provide some context for how fast it is compared to standard diffusion. We do our comparison using a batch size of $1$ to generate $10$ images with $50$ denoising steps. CPSample with $\alpha=0.5$ (i.e. computing gradients of the classifier at every step) had an average per-image generation time of $26.1\pm 0.029s$. By contrast, standard stable diffusion had an average generation time of $23.92\pm 0.055s$. Therefore, when the classifier is small compared to the size of the diffusion model, the added time cost is insignificant.

## Appendix E Membership Inference Attacks

Figure: Algorithm 1 Test statistic for membership inference attack against diffusion models [matsumoto2023membership]

In keeping with our goal of preventing membership inference attacks that are based on high similarity to a single member of the training set, we also perform a permutation test to ensure that we are not producing images that are anomalously close to the training data. Explicitly, we test the null hypothesis: generating images from CPSample produces images that are no more similar to the training data than they are to arbitrary points drawn from the data distribution. Our tests are performed in the same setting used in Section 4.1.
Let $S=\{x_{1},...,x_{k}\}$ be the data used for fine-tuning. Let $T=\{x_{1},...,x_{k},x_{k+1},...,x_{n}\}$ be the entire training set. Finally, let $P=\{\tilde{x}_{1},...,\tilde{x}_{m}\}$ be samples from CPSample. Then our permutation test is as follows:

- 1.
Sample $\tilde{x}_{1},...,\tilde{x}_{k}$ from $P$ without replacement. For each $\tilde{x}_{i}$, compute the quantity in 2.9 where the nearest neighbor is chosen among $S$. Let the similarity score of the most similar pair be $a$.
- 2.
Repeat the following process $\ell$ times: sample $S^{i}\subset T$ without replacement from $T$ so that $|S^{i}|=k$. Sample $P^{i}$ without replacement from $P$ so that $|P^{i}|=k$. Compute the most similar image in $S^{i}$ for each member of $P^{i}$. Call the similarity of the most similar pair $a_{i}$.
- 3.
For a pre-specified level $\alpha$, reject the null hypothesis if $\frac{1}{\ell}\sum_{i=1}^{\ell}\mathbf{1}\{a_{0}>a_{i}\}>\alpha$.

The results can be found in Table 6. Note that the test fails to reject on CIFAR-10 and LSUN Church, but succeeds on CelebA. This is likely because we fine-tuned the CelebA model more extensively than the other two.

**Table 6: Reduction in cosine similarity between generated images and nearest neighbor in fine-tuning data.**
| Dataset | FT Steps | $\alpha$ | Scale | DDIM | CPSample |
| --- | --- | --- | --- | --- | --- |
| CIFAR-10 | 150k | 0.001 | 1 | 0.92 | 0.47 |
| CelebA | 650k | 0.001 | 1000 | 0.99 | 0.99 |
| LSUN Church | 455k | 0.1 | 10 | 0.99 | 0.60 |

## Appendix F Additional Empirical Results

**Table 7: FID score *w.r.t.* $\alpha$ and Scale on CIFAR-10.**
|  | $\alpha=0.001$ | $\alpha=0.01$ | $\alpha=0.1$ | $\alpha=0.25$ | $\alpha=0.49$ |
| --- | --- | --- | --- | --- | --- |
| $\text{Scale}=1$ | 4.14275 | 4.15467 | 4.19058 | 4.19208 | 4.21859 |
| $\text{Scale}=5$ | 4.15772 | 4.20731 | 4.36005 | 4.58839 | 4.9566 |
| $\text{Scale}=10$ | 4.18083 | 4.26594 | 5.05858 | 6.17326 | 7.88949 |
| $\text{Scale}=100$ | 4.96727 | 16.7173 | 74.7247 | 113.199 | 139.626 |

Figure: Figure 9: Similarity scores with nearest neighbor for standard DDIM and CPSample ($\alpha=0.1$, scale$=10$) on LSUN Church. In both cases, the network was fine-tuned for 455k gradient steps on a subset of $1000$ images.
Refer to caption: x3.png

Figure: Figure 10: Uncurated samples using standard DDIM fine-tuned for 455k gradient steps on a subset of 1000 images from LSUN Church.
Refer to caption: extracted/5844491/church_unprot.png

Figure: Figure 11: Uncurated samples using CPSample ($\alpha=0.1$, scale$=10$) applied to a network fine-tuned for 455k gradient steps on a subset of 1000 images from LSUN Church. Note that there is no visual discrepancy in quality between these and the images from standard DDIM.
Refer to caption: extracted/5844491/LSUN_church_10_10.png

Figure: Figure 12: Uncurated samples using standard DDIM fine-tuned for 580k gradient steps on a subset of 1000 images from CelebA.
Refer to caption: extracted/5844491/Celeba_DDIM.png

Figure: Figure 13: Uncurated samples using CPSample ($\alpha=0.001$, scale$=1000$) applied to a network fine-tuned for 580k gradient steps on a subset of 1000 images from CelebA. Note that there is little visual discrepancy in quality between these and the images from standard DDIM.
Refer to caption: extracted/5844491/celeba_1000_protected.png

Figure: Figure 14: Uncurated samples using standard DDIM fine-tuned for 150k gradient steps on a subset of 1000 images from CIFAR-10.
Refer to caption: extracted/5844491/cifar_orig.png

Figure: Figure 15: Uncurated samples using CPSample ($\alpha=0.001$, scale$=1$) applied to a network fine-tuned for approximately 150k gradient steps on a subset of 1000 images from CelebA. Note that there is little visual discrepancy in quality between these and the images from standard DDIM.
Refer to caption: extracted/5844491/cifar_0001_1.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. As mentioned in the first point of weaknesses, it would be great if the authors could elaborate more on the the third assumptions either empirically or theoretically along with the $\delta$ problem mentioned.
soundness: 3
strengths: 1. The proposed method presents interesting facet and novelty where a classifier is utilized in remembering the statistics of training distribution and acts as guidance to steer away from the training data.
2. The planned experiments are great in terms of studying the effect of CPSample in a holistic manner. The three different experiments demonstrate different aspect of CPSample which is helpful for understanding the empirical effect.
summary: The paper proposes Classifier-Protected Sampling (CPSample), a novel technique to prevent diffusion models from generating training data  duplicates, apart from previous methods such as rejection sampling or other sampling methods that occurs during training phase, CPSample allows the diffusion model to steer away from generating training samples during test time, resolving the problem of privacy leakage.
weaknesses: 1. The presented theoretical study seems to be not proven much in terms of CPSample. Specifically, it is not demonstrated as to why assumption 3 holds well empirically as stated in the paper. Furthermore, given the intractable estimation and sometimes large values of Lipschitz constant, I wonder if that would make the possible $\delta$ extremely small which in terms renders the theory results in vain?
2. Some more experiments on MIA could be carried out to effectively evaluated the information loss caused when CPSample. Particularly, papers such as [1, 2] have devised new MIAs tailored to the settings of diffusion models which also conforms to the past settings in MIA.  It would be great to see if CPSample could also withstand these newly devised attacks.



[1]: Duan, Jinhao, et al. "Are diffusion models vulnerable to membership inference attacks?." International Conference on Machine Learning. PMLR, 2023.
[2]: Kong, Fei, et al. "An efficient membership inference attack for the diffusion model by proximal initialization." arXiv preprint arXiv:2305.18355 (2023).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1.	The abstract states: “CPSample achieves FID scores of 4.97 and 2.97 on CIFAR-10 and CelebA-64, respectively, without producing exact replicates of the training data.” What is the baseline? What model was it checked on? Without this background and some reference, these results do not have any meaning.
2.	The paper should cite another work that applied DP to diffusion models [1].
3.	Introduction: this method “fortifies against some membership inference attacks” This statement is rather too vague and it should be clearly stated what the goal of the method is. The defenses should not prevent some attacks that can infer the training images because the adversary would simply use the attacks that work and are not prevented by the proposed method.
4.	The fact that better FID scores are achieved than other methods has to be presented with some reference. Is it the trade-off that the higher performance stems simply from the worse protection?
5.	The background of the paper takes 2 pages. It is way too long and should be limited to a single page. For example, DP is not used in this paper so there is no need to provide this background section. Moreover, providing the formula for cosine similarity is really too trivial. The statement about the empirical results from this paper in Section 2.2 is very surprising: “Empirically, for CIFAR-10, we observed that images with similarity scores above 0.97 were nearly identical, whereas for CelebA, the threshold was approximately 0.95, and for LSUN Church, images with similarity above 0.90 were sometimes, though not always, nearly identical.” The same is repeated in Section 4.1.
6.	The definition of Differential privacy does not come from Dwork & Roth 2014. The first paper that proposed the mathematical framework was [2].
7.	Why is only a single membership inference attack used by Matsumoto et al. 2023a? The latest “evaluation of state-of-the-art MIAs on diffusion models reveals critical flaws and overly optimistic performance estimates in existing MIA evaluation” [3]
8.	Why are only the binary labels used and not more labels in the CPSample?
9.	From Section 4.1: Figure 1 does not show the most similar pairs of samples and fine-tuning data points.

**References:**

[1] dp-promise: Differentially Private Diffusion Probabilistic Models for Image Synthesis. Haichen Wang, Shuchao Pang, Zhigang Lu, Yihang Rao, Yongbin Zhou, Minhui Xue. USENIS 2024.

[2] Calibrating noise to sensitivity in private data analysis. Dwork, C., McSherry, F., Nissim, K., & Smith, A. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006. 

[3] Real-World Benchmarks Make Membership Inference Attacks Fail on Diffusion Models. Chumeng Liang, Jiaxuan You 2024: https://arxiv.org/abs/2410.03640
soundness: 1
strengths: 1. The paper tries to solve a very important problem: how to prevent the diffusion models from generating the training samples.
summary: The paper proposes a method to prevent the generation of training data during the inference on diffusion models. The way to obtain the result is by modifying the sampling process in diffusion models. The method leverages a classifier that is trained and forced to overfit on random binary labels assigned to the training data of diffusion models. The method uses “classifier guidance” to prevent the generation process from returning the training data points.
weaknesses: 1.	This method is very simple: “Overfit a classifier on random binary labels assigned to the training data and use this classifier during sampling to guide the images away from the training data.” Does it still require training the classifier on the large training dataset of diffusion models? What if we do not have access to the training data any longer?
2.	As stated in Section 2.1: “Currently, the state-of-the-art for guided generation is achieved by models with classifier-free guidance (Ho & Salimans, 2022). However, since CPSample employs a classifier to prevent replication of its training data,” Thus, the CPSample method is not applicable to the SoTA diffusion models. 
3.	The proposed CPSample method is very similar to the AMG guidance method from Chen et al. 2024. It is claimed that AMG requires access to the training data of a diffusion model. How is the classifier trained for the proposed CPSample method without requiring access to the training data? (as stated at the end of Section 3.1)
4.	The qualitative results in Figure 3 clearly show the degradation of quality in the generated images between DDIM and CPSample (especially for the LSUN Church dataset). Additionally, Image C in the last row for the CPSample shows the complete degradation in the quality of the generated images. 
5.	What is the overhead of the method in terms of the inference time? How long does it take to train the classifier? How is the classifier trained? 
6.	The paper only compares the FID score of the CPSample to other methods, however, no comparison is presented in terms of the reduction in the generation of the training data points.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. In my opinion, the trained overfitted classifier is designed to estimate a memorization score and captures a boundary between the member set and hold-out set to provide a guided gradient. However, various alternative methods exist for estimating this boundary or providing a similar memorization gradient. For instance, training a classifier to differentiate between the member set and hold-out set, or training an overfitted diffusion model to provide memorization gradients, are both plausible approaches. Therefore, why choose to assign random 0/1 labels to the member set? (The reasons provided in Line 238-244 is not very convincing for me) 

2. The scalability of CPSample to diffusion models with larger member sets is uncertain. Current experiments only provide limited examples (Section 4.2). Additional examples are required to further demonstrate the effectiveness of the proposed method. Furthermore, the hyperparameters ($\alpha$, $s$) for CPSample seems dependent on each sampling. How to define these hyperparameters for each sampling? 

3. The experiments on similarity reduction have been conducted on low-resolution and small-scale datasets such as Cifar10 and CelebA. For these datasets, the diffusion model can verbatim memorize each pixel of the training samples. However, in more practical scenarios, diffusion models tend to memorize only the rough structure of the training sample and can only reproduce similar patterns. Consequently, the current experiments (Section 4.1) may not adequately demonstrate the effectiveness of the proposed method for similarity reduction. Additional experiments on large-scale diffusion models are necessary to further validate the method.

4. The current experiments have been mostly conducted on unconditional diffusion models. To fully evaluate the effectiveness of CPSample, additional experiments on conditional diffusion models, such as text-to-image diffusion models, should be conducted to assess whether CPSample can accurately align with the given condition.
soundness: 3
strengths: 1. The issue of replicate generation in diffusion models is critical, and there is an urgent need for a scalable solution to address it.
2. The concept of explicitly training a classifier to measure the memorization score is novel, and subsequent experiments have demonstrated its efficacy.
summary: Diffusion models exhibit a tendency to memorize training images and reproduce these training images during inference, which can pose significant privacy concerns. To address this issue, this paper introduces a sampling strategy (CPSample) guided by a trained classifier. This classifier evaluates the memorization score of current generated image and rectifies the sampling trajectories to diverge from the training images. By adjusting the guidance scale, the proposed method ensures that the generated images possess a memorization score approximating 0.5, thereby mitigating the risk of replicate generation.
weaknesses: 1. The scalability of the strategy to train a classifier is a primary concern. Given that current text-to-image diffusion models, such as Stable Diffusion and Pixart, are trained on large-scale datasets scraped from the Internet, is it a feasible solution to train an overfitted classifier on a large scale dataset? 

2. The proposed method assumes that the training member set is available. However, this assumption does not always hold. Specifically, the training member sets of certrain text-to-image diffusion models, such as Pixart, are not publicly accessible, therby limiting the applicability of the proposed approach.

3. While the proposed CPSample method effectively mitigates replicate generation, it also exhibits side effect on the generated images as it manipulates the sampling trajectory. For example, in Figure 4, the generated image by CPSample does not align with the input prompt and sometimes will generate corrupted results. In contrast, although rejection sampling faces challenges with resampling, it tends to produce images that are more closely aligned with the input prompt. 

4. There are many typos in notations of DDPM, for example, $\alpha_t \rightarrow \bar{\alpha}_t$ in Equation 2.1&2.2, Line 146. Please check this through the whole manuscripts to ensure the consistency and accuracy of all notations.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1) Can you show how your method performs on large-scale datasets?
soundness: 3
strengths: 1) The method is sound and solid. It implements the idea of rejection sampling by using classifier guidance. This achieves training-free prevention of training data replication.

2) Experiments show that the proposed method can avoid training data replication even if there training setup leads to potential over-fitting. This proves the effectiveness of the proposed method,
summary: This paper proposes a rejection sampling method that avoids diffusion models from replicating training data in generated contents. The main idea is to use classifier guidance dependent  on the label of training data / hold-out data. This gets the sampling process away from the region of training data and then preventing diffusion models from generating training data.
weaknesses: 1) Motivation: the main weakness of this paper lies in its motivation. As shown in existing research, training data replication is not a common phenomena in real-world pre-trained diffusion models. Most training samples are even unable to be inferred as member by membership inference attacks in Stable Diffusion, while the exception ones can be handled case by case. While the proposed method still degrades the generation performance as well as the diversity (potentially), it seems uneconomical to deploy the proposed method on any pre-trained diffusion models. Since privacy issues only happen in pre-trained diffusion models, the significance of the proposed method is moderate.

2) Computation cost & Generalizability: As stated in the paper, we need to train a classifier on all training and some hold-out examples. This could be significantly computationally costly without promising generalizability, since no one has trained such a classifier before. In fact, the training of the classifier in the paper only includes less than 2 thousands images. However, the training datasets of pre-trained diffusion models consist of billions of images. It leaves doubts whether a classifier can effectively give guidance based on these large-scale datasets.

3) OOD generation (minor): One potential drawback, as shown in Figure 4, is that the proposed method may generate images with contents quite different to the description of the text.

In general, although the proposed method is interesting and performs soundly on toy diffusion models, it does not seem to practical to deploy it on real-world AIGC applications which raises the data privacy concerns. Hence, I do not recommend accepting this paper.

Meta-review: This paper studies the problem of alleviating privacy leakage in diffusion models, which occurs when these reproduce samples very close to those in training data and membership inference attacks. This is an important problem, given that state-of-the-art models are often trained in private or copy-righted data. The authors' proposed approach consists of fitting a classifier on the samples to be protected with random binary labels, and then employing it to steer the diffusion process away from those samples at inference time. While empirical and not endowed with theoretical guarantees (like those in differential privacy), experiments show that the approach effectively diminishes the reproduction of training samples, while compromising the sampling quality only mildly.

**Strengths**
* The problem studied is very relevant to modern SOTA diffusion models.
* The presented approach is conceptually simple.
* Their approach does not require re-training the diffusion model, potentially applying to off-the-shelf models,

**Weaknesses**
* The motivation of the work, and conceptual advantages w.r.t. other methods, could be presented more clearly.
* The need to train a classifier to 'memorize' the protected data could be a significant computational overhead.

**Summary**
In all, this paper provides a potential solution to the problem of reproduction of training samples from diffusion models, including SOTA models like Stable Diffusion. This is a hard problem, and the method does a reasonable job at a new, simple, and innovative solution. Most reviewers agree that the approach is effective in accomplishing this task.
The computational overhead, and the need to have access to the data to protect, seems almost unavoidable, and it might be cost to pay for protecting the private data. A significant amount of discussion was entertained by the reviewers, who are mostly leaning to reject - see further comments below. However, upon carefully all comments, responses, and the paper, this AC thinks that the strengths outweigh the weaknesses of this paper, and the recommendations from the reviewers fail to recognize this. I am therefore recommending acceptance with the following recommendations:

* Motivation: most reviewers were concerned with the need to have access to training data. In the discussions, the authors stressed that access to all of the training data might not be needed, (just) access to the data to be protected (a subset) is needed. Please make this motivation clear, positioning this into a practical scenario (as argued in your responses).
* Computational overhead: explain how and why the computational and memory overhead of your method is comparable to other approaches (as argued in your responses).
* Improve the presentation by incorporating new references and comments, where appropriate (as suggested by reviewers).
* Theoretical basis: Please stress the limitations of the provided theoretical results, as well as the limitation of this method in providing theoretical guarantees against privacy leakage (unlike, say, DP).
* Comment on the AMG method referenced by 8m3n, and their differences.
* Stress on how the quality of your produced samples and compare to those in competing methods.
* Stress on the limitations of the assumption of the model size scaling linearly with the number of samples to be protected, as mentioned by Rev. t8KP. I agree with the authors that, in the scale studied here (up to 160K), this seems to be feasible, but it is unclear if it will remain feasible in one or two orders of magnitude higher.

Decision: Accept (Poster)