Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces differential private diffusion models (SPDMs) and studies DP-SGD for DPDMs training. This paper proposes a modification of the diffusion model training objective called noise multiplicity to boost performance. The DPDMs experiments show improvement in image generation tasks. 
strength_and_weaknesses: Strengths:

1. Studying differential privacy for the diffusion model is a good initiation. 
2.The noise multiplicity idea for a less noisy objective function is new and interesting. 
3.This paper also provides positive experiment results using DPDMs. 

Weakness

1. The motivation for connecting differential privacy with diffusion models is not well explained.  The arguments in section 3.1 do not fully explain the advantage of diffusion models for differential privacy. The privacy guarantee of DPDMs comes from DP-SGD training which is not new. It would be interesting to show the diffusion model itself benefits privacy.
2. The advantage of DP-SGD for differential private diffusion models is not well supported. The paper claims that the DP generative model is challenging due to injected noise in training. But DPDMs proposed in this paper also uses DP-SGD for training which also injects noise into training. It will be more convincing if the paper can add theoretical or experimental support that DP-SGD injects less noise for DPDMs than for other generative models 
3. It will be helpful to have a theoretical or empirical explanation that noise multiplicity reduced the noise injected in the training. It also will be interesting if the paper empirically compares the noise variance in the gradient with different levels of K. Also, the paper can clarify if noise multiplicity is unique to diffusion models or if this idea can be generalized to other generative models.

clarity,_quality,_novelty_and_reproducibility: The clarity and novelty of this paper are moderate. The experiments could be reproducible with the details provided in this paper.
summary_of_the_review: I feel the approach proposed in this paper is not well motivated. The significance needs to be improved. Overall, I think this submission is not ready for ICLR. Thus, I recommend rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studied the differential privacy preservation for the generative model and proposed incorporating the DP-SGD strategy into Diffusion Model training. 
strength_and_weaknesses: Strength:
1. This paper is well-organized and easy to follow.
2. The experimental results on MNIST, Fashion-MNIST, and CelebA datasets validate the effectiveness of the proposed method, and DPDM achieves state-of-the-art performance.

Weaknesses:
1. This paper seems simply adopt the well-established DP-SGD into Diffusion Model training. The novelty seems limited.
2. The motivation of the DPDM is somewhat weak, there exist various generative models, such as GAN, VAE, Diffusion, and so on. Training a GAN is harder than VAE and Diffusion model. However, previous works [G-PATE, PATE-GAN, DPGAN, GS-WGAN] demonstrate the effectiveness of training a DP-GAN-like model. 
3. The datasets adopted have small resolutions. For example, the authors use $28 \times 28$ resolution for MNIST and Fashion-MNIST, $32 \times 32$ resolution for CelebA dataset. Training a high-resolution model might be realistic in the application, as the Diffusion Model is well-known for its outstanding performance in high-resolution image synthesis. It might be hard for DP-SGD to train with high resolution while perverse the utility of the outputs.

clarity,_quality,_novelty_and_reproducibility: The paper is well-structured and clearly written. The definition of the statistical method is easy to follow.
The novelty of the paper is somewhat incremental.
The reproducibility of the paper seems good as relevant details as listed in the paper.
summary_of_the_review: The main concern of the reviewer is the novelty of DPDM. It seems that the DPDM is a combination of DP-SGD and DDIM models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper trains tailored diffusion models with differential privacy. Two intuitive motivations for using diffusion models for DP data generation are 1) the loss function is a simple and scalable $L_2$ regression loss; 2) the denoiser network in diffusion models is simpler and smoother because it is not designed to generate data in a one-shot fashion. Several novel modifications on the design choices of diffusion models are proposed to boost the performance. One worth mentioning modification to the objective function of non-private diffusion models is *noise multiplicity*. At each step, the authors sample $K$ noises and take the average of the gradients of $K$ denoising losses. Experiments show the proposed method achieves SOTA performance.
strength_and_weaknesses: **Strength**

1. This is the first work to use diffusion models for DP data generation. The authors achieve state-of-the-art performance on several image benchmarks. 

2. The authors propose several novel modifications to the non-private implementation of diffusion models, which are necessary to achieve good performance.

3. The paper is well written. I enjoy reading it.

**Weaknesses**

I have one concern about the benchmarks used in this paper. They have a common pattern: every image has very visually similar neighbors. For MNIST, all digits from the same class look very similar. For CelebA, there are dozens of images for most of the identities. I understand these two datasets are standard in the literature, but I think they are not representative enough. My suggestion is to include some other datasets that do not have clear clusters in the raw input space, e.g., CIFAR-10 which is also used in some previous works. I don't expect the results to be as good as those in the current version, but it is good for the community to know the results. 

Another small concern about CelebA is that the privacy guarantee is example-wise. Each identity has dozens of examples so the identity-wise privacy guarantee will be loose. Again, I understand this is the benchmark from previous works, but still want to express my concern here.
clarity,_quality,_novelty_and_reproducibility: Please see Strength And Weaknesses.
summary_of_the_review: I recommend acceptance because this paper gives a high-quality implementation of DP diffusion models and demonstrates promising results. I hope the authors could reply to my first concern.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to use DP-SGD for training the score function in training diffusion models (DM), and proposes noise multiplicity to improve the SNR in the cost  estimation for training the DM. 

The main strength in the paper is that shows on image datasets that the DP models quality is improved with respect to previous work. 

The main weaknesses of the work is the  novelty of  the work, as it  builds on DP-SGD and noise multiplicity previously proposed in De et al. Another weakness is that the evaluation is only on image quality / accuracy of downstream task on DP synthetic data and there is not any metric on the privacy of the obtained models. 


justification_for_why_not_higher_score: The paper novelty is limited and the evaluation on the privacy of the resulting models/ synthetic data is not provided.
Decision: Reject