On Memorization in Diffusion Models
Authors: Xiangming Gu
Chao Du
Tianyu Pang
Chongxuan Li
Min Lin
Ye Wang
Venue: Accepted by TMLR ()

Abstract: Due to their capacity to generate novel and high-quality samples, diffusion models have attracted significant research interest in recent years. Notably, the typical training objective of diffusion models, i.e., denoising score matching, has a closed-form optimal solution that can only generate training-data replicating samples. This indicates that a memorization behavior is theoretically expected, which contradicts the common generalization ability of state-of-the-art diffusion models, and thus calls for a deeper understanding. Looking into this, we first observe that memorization behaviors tend to occur on smaller-sized datasets, which motivates our definition of effective model memorization (EMM), a metric measuring the maximum size of training data at which a model approximates its theoretical optimum. Then, we quantify the impact of the influential factors on these memorization behaviors in terms of EMM, focusing primarily on data distribution, model configuration, and training procedure. Besides comprehensive empirical results identifying the influential factors, we surprisingly find that conditioning training data on uninformative random labels can significantly trigger the memorization in diffusion models. Our study holds practical significance for diffusion model users and offers clues to theoretical research in deep generative models.

Published in Transactions on Machine Learning Research (02/2025)
On Memorization in Diffusion Models
Xiangming Gu'
riangrrinng Cu. mus. edu
National University of Singopore
Chao Du Sea Al Lab
duchao Osca, com
Tianyu Pang Sea Al Lab
Maanyuuann@@scc... com
Chongxuan Li
chongpuaniirruc.ctu.cmm
Gaoling School of Artificial Intelligence Renmin University of China
Min Lin Sea AI Lab
linmin Saamin&@sea.com
Ye Wang!
wangye@comp..n edu.sg
National University of Singapore
Reviewed on OpenReview: https: // openreview. net/ forum? id= D3DBqu
Abstract
Due to their capacity to generate novel and high-quality samples, diffusion models have attracted significant research interest in recent years. Notably, the typical training objective of diffusion models, i.c., denoising score matching, has closed-form optimal solution that can only generale training-data replicating samples. This indicates that a memorization behavior is theoretically expected, which contradicts the common generalization ability of stato-of-the-art diffusion models, and thus calls for deeper understanding. Looking into this, we first observe that memorization behaviors tend to occur on smaller-sized datasets, which motivates measuring the maximum size of training data at which model approximates its theoretical optimum. Then, we quantify the impact of the influential factors on these memorization behaviors in terms of FMM, focusing primarily on data distribution, model configuration, and training procodure. Besides comprchensive empirical results identifying the influential factors, we surprisingly find that conditioning training data on uninformative random labels can significantly trigger the memorization in diffusion models. Our study holds practical significance for diffusion model users and offers clues to theoretical rescarch in deep generative models. Code is available at
definition of effective model memorization (EMM), metric
com http:/aallaggDiifMemniiiii
1
Introduction
In the last few years, diffusion models Sohh-Dickstein et al.
2015 Song & Ermon 2019 Ho et al. 2020 Song
el al. 2021b) have achieved significant success across diverse domains of generalive modeling, including image
generation (Dhariwal Nichol 2021 Karras et al.] 2022).
sett-to-image synthesis (Rombach et al. 2022
Ramesh et al. (022), audio speech synthesis (Kim
2022 Huang et al. 2023). graph generation Xuu
2022 ignac et al. 2022 and 3D content generation (Poole et al.) 2023 Lin et al. 2023 Substantial et al.
"Work doue during internship Sea Al Lab.
1 &Corresponding authors.

---

Published in Transactions on Machine Learning Research (02/2025)
(b)
16
100
Ratio (%) of Memorization
14
[D]
Ratio (%) of Memorization
90
[2]
D
80
2k
12
10k
70
10
60
|D]
[m]] = 5k
50
40
30
20
10
0
0.8k
1.6k
2.4k
3.2k
4.0k
8k
16k
24k
32k
40k
Training Epochs
Training Epochs
(c)
(d)
Figure 1: Overall motivation. Generated images (top row) and their Ez-nearest training samples in D (boltom
row) by (a) the theoretical optimum defined in equation 2 (b) EDM (Karras et al., 2022). Memorization
Ratios (%) of EDM models trained with different [D]: (c) within 4k training epochs; (d) when extending to 40k training epochs.
empirical evidence altests to the ability of diffusion models to generate diverse and novel high-quality samples (Dhariwal Nichol 2021 Nichol & Dhariwal 2021 Nichol et al. 2021), underscoring their powerful capability of abstracting and comprehending the charactcristics of the training data.
Diffusion models posit forward diffusion process {=+), E T)) that gradually introduces Gaussian noise to data point resulting in transition distribution q( (()ll)) [2 [[z]a,,,, o.I). The coefficients a and are chosen such that the initial distribution 90(2)) aligns with the data distribution P(x) while steering it towards an approximately Gaussian distribution (r())) Sampling from the data distribution can then be achieved by reversing this process, for which a critical unknown term is the data score V., log gc(2)) (Song et al. 2021b). Diffusion models approxinate the data scores with a score model 89 (z t), which is typically learned via denoising score matching (DSM) [Vincent 2011
N
(D)
A
Σ R=1
Byc
so(aro + o2c,t) + σt
(1)
2N
given NO, I) and a dataset of N training samples a (1x, to identify the optimal solution of equation |
p(x) Interestingly, it is difficult (assurning sufficient capacity of 0, see proof in Appendix A.1
Σ exp (- |aaran' 2ell2
|lartn -2el2
8 (=t,
Σ exp
atTT 2t of
(2)
207
207
which, however, lcads the reverse process towards the empirical data distribution, defined as P(x) N EA== 6(x - x..) Consequently, the optimal score model equation can only produce samples that replicate the training data, shown in Figure [(a] suggesting a memorization behavior (van den Burg & Williams 2021 This evidently contradicts the typical generalization capability exhibited by state-of-the-ar.
diffusion models such EDM [Krrass et al. 2022). illustrated in Figure I(b)
Such intriguing gap prompts inquiries into (i) the conditions under which the learned diffusion models can faithfully approximate the optimum 8* (essentially showing memorization) and (ii) the influential
We also provide& theoretical analysis from the lens backward process Appendix A.2
2

---

Published in Transactions on Machine Learning Research (02/2025)
factors governing memorization behaviors in diffusion models. Besides clear issue of potential adverse generalization performance (Yoon et al. 2023]; it further raises a crucial concern that diffusion models trained with equation [ might imperceptibly memorize the training data, exposing several risks such as
privacy leakage (Somopalli et al. 2023b) and copyright infringement (Somepalli et al.] 2023a Zhao et al.)
2023 Wang et al., 2024]. For example, Carlini et al. (2023) show that is possible to extract a lew training images from Stable Diffusion (Rombach et al., 2022)
substantiating a tangible hazard.
In response to these inquiries and concerns, this paper presents comprehensive empirical study on mem- orization behavior in widely adopted diffusion models, including EDM (Karras et al., 2022) and Stable Diffusion (Rombach et al. (022]. We start with an analysis of EDM, noting that memorization tends to occur when trained on smaller-sized datasets, while remaining undetectable on larger datasets. This motivates our definition of effective model memorization (EMM), a metric quantifying the maximum number of training data points (sampled from distribution P) at which a diffusion model M demonstrates the similar memorization behavior as the theoretical optimum after the training procedure T. We then quantify the impact of critical factors on memorization in terms of EMM the CIFAR-10, FFHQ (Karras al. 2019 and Imagenette (Deng et al.. 2009 Somepalli et al. 2023b) datasets, considering the three facets of P, M,, and T. Among all illuminating results, we surprisingly observe that the memorization can be triggered by conditioning training data on completely random and uninformative labels. Specifically, using such conditioning design, we show that more than 65% of samples generated by diffusion models trained on the 50k CIFAR-10 images are replicas of training data, an obvious contrast to the original 0%. Our study holds practical significance for diffusion model users and offers clues to theoretical research in deep generative models.
2 Memorization in Diffusion Models
We start by examining the memorization in the widely-adopted EDM (Karras et al. 2022). which is one of the stato-of-the-art diffusion models for image generation. To determine whether generated image a is a memorized replica from the training data D, we adopt the criteria. introduced in Yoon et al. (2023). which considers x as memorized if its &g distance to the nearest neighbor is smaller than 1/3 of that to the second nearest neighbor in the training data. Here the factor 1/3 is an empirical threshold as it accurately aligns human perception of memorization (Yoon et al. 2023). We train an EDM model on the CIFAR-10 dataset without applying data augmentation (to avoid any ambiguity regarding memorization) and evaluate the ratio of memorization among 10k generated images. Remarkably, we observe a memorization ratio of zero throughout the entire training process, as illustrated by the bottom curve in Figure [[c]
Intuitively, we hypothesize that the default. configuration of EDM lacks the essential capacity to memorize the 50k training images in CIFAR-10, which motivates our exploration into whether expected memorization behavior will manifest when reducing the training dataset size. In particular, we generate a sequence of training datasets with different sizes of {20k, 10k, 5k, 2k), by sampling subsets from the original set. of 50k CIFAR-10 training images. We follow the default EDM training procedure (Karras et al., 2022) with consistent training epochs on these smaller datasets. As shown in Figure [[c] when [D] = 20k or 10k, the memorization ratio remains close to zero. However, upon further reducing the training dataset size to [D] = 5k or 2k, the EDM models exhibit noticeable memorization. This observation indicates the substantial impact of training datasct size on memorization. Additionally, we notice that the memorization ratio increases with more training epochs. To observe this, we extend the training duration to 40k epochs, ten times longer than that in Karras et al. (2022). In Figure [[d] when [D] = 1k, the model achieves over 90% memorization. However, even with 40k training epochs, the diffusion model still struggles to replicate a large portion of training samples when 5k.
Based on our findings, we seek to quantify the maximum dataset size at which diffusion models demonstrate behavior similar to the thcoretical optimum, which is crucial in understanding memorization behavior. To formalize this notion, considering a data distribution P, a diffusion model configuration M, and a training procedure T, we introduce the concept of effective model memorization with the definition as follows inspired by Yoon et al. (2023).
3

---

Published in Transactions on Machine Learning Research (02/202))
Ratio (%) Memorization
Ratio (%) of Menoriztioon
100
Ratio (%) Memorization
100
90
80
90
90
60
90
70
70
--
40
30
Ros 32X32
C
20
Ros 16x
50
Cm2
50
0.2
10
Res 8X3
C=1
== 0.0
0.5k
1k
2k
4k
8k
16k
40
0.5k
40
1k
2k
Data Size
[D]
Data Size |D|
Data Size [D]
(a) Varying data dimensions
(b) Varying inter-diversity
(c) Varying intra-diversity
Figure Memorization ratios (%) of diffusion models on CIFAR-10 under different factors of data
distribution P. The intersections of dashed line (90%) and different curves the estimates of EMMss
Definition 2.1 (Effective model memorization). The effective model memorization (EMM) with respect to P, M, T and parameter <> 0, is defined
(P, = max (
(3)
where RMem refers to the ratio of memorization.
EMM indicates the condition under which the learned diffusion model approximates the theoretical optimum and reveals how P, M, and interact and affect memorization. Our definition assumes that a higher memorization ratio tends to occur smaller-sized training datasets, which is stated
Hypothesis 2.2. Given M, T and two training datasets D1 and D22 both of which are sampled from the same data distribution P, the ratio of memorization satisfies
RMem (Di, M, > RMem (D2, if if C D2 and D1, ~.
(4)
Based on Hypothesis 2.2 we provide a viable way to estimate EMM. Specifically, we sample a series of training datasets D1, D2...., with different sizes from the data distribution P, and then train diffusion models with configuration M following the training procodure T. Afterwards, we evaluate the ratio of memorization RMem (D1, M,T), RMem MD,
and then determine the size of training dataset which meets that RMem MM, T) == 1 5. We note that it is computational intractable to determine the accurate value of interpolate the value of EMM based on two consecutive sampled datasets D,, Di+1 that
EMM. Therefore,
RMem (D, > 1 - and RAte (D1+1, < - Therefore, this study is formulated as how the above three factors P, M, and T affect the measurement of EMM. There is no principled way to select the value of (, so we set it as 0.1 based on our experiments Figure [(d] throughout our study. We include an analysis on the sensitivity of Ç in Appendix
We set CIFAR-10 our default dataset, and the basic experimental setup introduced in Appendix B which is well-adopted recipe for diffusion models. We highlight that compared to Karras et al. (2022) run 10 times the number of training epochs. The experiments related to FFHQ [Karras et al.] 2019) and
Imagenette [Deng et al. 2009 Somepalli et al.) 2023b) are shown in Appendix D and Section 6 respectively.
For evaluation, we also consider the image quality metric, e.g., Fréchet Inception Distance (FID) (Heusel et al. 2017) in Appendix that when diffusion models memorize a significant proportion of training data, the metrics related image quality (including FID) are also close to optimal. This intuition is in line with our experimental results in Appendix E Morcover, when considering alternative memorization metrics, the relationship of various factors and memorization remains consistent.
and alternative memorization metric in Appendix F It is worth mentioning
3 Data Distribution P
In the preceding section, we have illustrated the substantial impact of the size of training data on the memorization in diffusion models and how evaluate the value of effective model memorization (EMM).
4

---

Published in Transactions on Machine Learning Research (02/2025)
We now proceed to investigate the influence of specific attributes of the data distribution P on the EMM, focusing primarily the dimensions and diversity of the data. We keep both the model configuration M and the training procedure T fixed throughout this section.
3.1 Data Dimension
As likelihood-based generative models, diffusion models could face challenges when fitting high-resolution images, stemming from their mode-covering behavior as noted by Rombach et al. (2022). To explore the influence of data dimensionality on the memorization tendencies of diffusion models, we evaluate the EMMs on C1FAR-10 at various resolutions: 32x32, 16x 16 and 8x8, where the latter two are obtained by
downsampling. Note that the U-Net (Ronneberger et al. 2015) seamlessly accommodates inputs of different resolutions, requiring modification of the model configuration M..
For each resolution, we sample a series of training datasets D with varying sizes and evaluate the memorization ratios of trained diffusion models. As illustrated in Figure ((a) we estimate the EMM for each resolution by determining the intersection between the line of 90% memorization (dashed line) and the memorization curve. The results reveal natural insights into the EMM with varying input dimensions. Specifically, for the 32x32 input resolution, we observe an EMM of approximately 1k. Transitioning to a 16x16 resolution, the EMM slightly surpasses 4k, while for the 8x8 resolution, it reaches approximately 8k. Furthermore, even for [D] 16k, the ratio of memorization still exceeds 60% when trained on 8x8 CIFAR-10 images. These results underscore the profound impact of data dimensionality on the memorization within diffusion models.
3.2 Data Diversity
Number of Classes. We consider four different data distributions by selecting C' E {1, 10} classes of images from CIFAR-10 and then evaluate the FMMss While varying the data size [D] during probing the EMMs, we ensure that each class contains an equal share [D|/C data instances. The results of EMMs for different C are shown in Figure 2(b] We find that as the number of classes increases, diffusion models tend to exhibit lower memorization ratio and lower EMM, which is consistent with the intuition that diverse data is harder to be memorized. Note, however, that this effect is subtle, as evidenced by the nearly identical EMMs observed for C' =5 and 10
Intra Class Diversity. We also explore the impact of intra- class diversity, which measures variations within individual classes. We conduct experiments with C 1, where only the Dog class of CIFAR-10 used. To control this diversity, we gradually blend images (scaled to a resolution of 32x32) from the Dog class in ImageNet (Deng et al. 2009) into the Dog class of CIFAR-10. We introduce an interpolation ratio, denoted as a (a E [0, 1]), representing the proportion of ImageNet data in the constructed training dataset. Notably, ImageNet's Dog class contains 123 sub-classes, indicating higher intra-class diversity compared to CIFAR-10. Consequently, larger corresponds to higher intra-class diversity. As shown in Figure 2(c) an increased blend of ImageNet data results in slightly lower EMM in the trained diffusion models. Similar to our experiments concerning the number of classes, these results reaffirm that diversity contributes limitedly to memorization.
4 Diffusion Model Configuration M
In this section, we study the influence of different model configurations on the memorization tendencies of diffusion models. Our evaluation encompassos several aspects of model design, such as model size (width and depth), the way to incorporate time embedding, and the presence of skip connections in the U-Net. Similar previous sections, we probe the EMMs by training multiple models on training data of different sizes from the same data distribution P (CIFAR-10), while keeping the training procedure T fixed.
4.1 Model Size
Diffusion models are usually constructed using the U-Net architecture (Ronneberger et al. 2015). We explore the influence of model size on memorization using two distinct approaches. First, we increase the channel multiplier, thereby augmenting the width of the model. Alternatively, we raise the number of residual blocks per resolution, as demonstrated by (Song et al. 2021b), to increase the model depth.
Model width. We explore different channel multipliers, specifically {128, 192, 256,3203, while keeping the number of residual blocks per resolution fixed at 2. As illustrated in Figure 3(a) it is evident that as the
5

---

Published in Transactions on Machine Learning Research (02/2023)
Ratio (%) of
100
(%K of Memorization
Ratio (%) of Memorizaiion
100
90
90
70
70-
Widd = 120x
70
Diget
40
PRRSS:
Widt
Dish
Peatir dipc
Wazt 251x
60
Dopak
Packnoal opt
Wist 2
Dapth
10x
10
Parkical, opp 4x
1K
2k
4k
5k
30
0.3k
1.0%
1.3%
2.0k
2.5k
3.0k
0.5k
Data Size |D|
Data Size
[D]
Data Size [P]
(a) Varying model widths
(b) Varying model depths
(c) Varying time embeddings
Figure Memorization ratios (%) of diffusion models on CIFAR-10 under different factors of model configuration M.
model width increases in diffuusin models, the EMMs exhibit monotonic rise. Notably, scaling the channel multiplier to 320 yields an EMM of approximately 5k, representing a four times increase compared to the EMM observed with a channel multiplier set at 128. These results show the direct relationship between model width and memorization in diffusion models.
Model depth. We vary model depth by adjusting the number of residual blocks per resolution (ranging from to 12), while maintaining a constant channel multiplier of 128. In contrast to the scenario of varying model width, modifying model depth yields non-monotonic effects
memorization. As present in Figure 3(b) the EMM initially increases as we scale the number of residual blocks per resolution from 2 to 6. However, when further increasing the model depth, the EMM starts to decrease. This non-monotomic trend is further shown in Figure 7(b)
We assess the above two approaches for scaling the model size of diffusion models. Model size scales linearly when augmenting model depth but quadratically when increasing model width. With channel multiplier set to 320, the diffusson model encompasses roughly 219M trainable parameters, yielding an EMM of approximately 5k. In contrast, with residual block number per resolution set to 12, the model contains approximately 138M parameters, resulting in an EMM only ranging between 1K and 2k. Further increasing the model depth may encounter failures in training. Consequently, scaling model width emerges as a more viable approach for increasing the memorization ratio of diffusion models. We conduct more experiments in Appendix C.1 to further confirm our conclusions.
4.2 Time Embedding
In our experimental setup (see Appendix Bh, employ the model architecture of DDPM+ (Song et al. 2021b;
Karras et al.) 2022), which incorporates positional embedding |Vaswani et al. 2017) to encode the diffusion ste. In addition to positional embedding, Song et al. (2021b) used random Fourier features (Tancik et al., 2020) in their NCSN++ models. Therefore, we conduct experiments to test both time embedding methods and assess their impact on memorization. As depicted FigureB(c) to further support our conclusion, we consider two diffusson models with and 4 residual blocks per resolution. We observe a significant decrease in the memorization ratio (and thus EMM) when using the Fourier features in DDPM++ highlighting the noteworthy effect of time embedding memorization. The same trend is observed on FFHQ (see Appendix D.2).
4.3 Skip Connections
We investigate the impact of skip connections, which are known for their significance in the success of U-Net (Ronneberger et al. 2015), on the memorization of diffusion models. Specifically, in our experimental setup (see Appendix BB, the number of skip connections is 3(n + 1), where n corresponds to the number of residual blocks per resolution. For each resolution, namely, 32x32, 16x16, 8x8, we establish + skip connections that bridge the output of the encoder to the input of the decoder. To conduct our experiments, we set the size of training dataset [D] to 1k, and the value of n to 2, resulting in a total of 9 skip connections in our model architecture.
Initially, we explore the influence of the quantity of skip connections on memorization. Intriguingly, our obser- vations reveal that even with a limited number of skip connections, the trained diffusion models are capable of maintaining memorization ratio equivalent that achieved with full skip connections, as demonstrated in
6

---

Published in Transactions on Machine Learning Research (02/2023)
100
70
80
B S S
(SQ Meneiia
Itaic cle of Merueeiaaoo
D. B.G 18 br 2- S.S.
50
S2K32
16Ki5 Spatal Benolatton
EXS
Indox of Skip Correction 3
31852
15X16
Bics
Soxial Roclltto
Index of Skip Cosnection
(a)
(b)
(c)
(d)
Figure 4: Memorization ratio (%) of diffusion models on CIFAR-10 when retaining (a) skip connections of certain spatial resolution for DDPM++; (b) single skip connection at different locations for DDPM++; (c)
skip connections of certain spatial resolution for NCSN++; (d) single skip connection at different locations
for NCSN++.
This underscores the role of skip connection sparsity significant factor influencing memoriza- tion in diffusion models, prompting our exploration of the effects of specific skip connections. In the following experiments, we consider both DDPM++ and NCSN++ architect in Song et al. (2021b); Karras et. al. (2022). Utilizing full skip connections, we observe that NCSN++ attains only approximately 45% memorization ratio, whereas DDPM++ achioves a memorization ratio exceods 90%, all under the identical training datasct 'D
Appendix C.2
Skip connection resolution. We narrow our examination to the inclusion of selected number of skip connections, specifically m 1,2,3, all situated at particular spatial resolution. The results, illustrated in Figure 4(a) and Figure [cc) unveil notable trends. We note that dilferent markers represent distinct quantities of skip connections. Our observations reveal that skip connections situated at higher resolutions contribute more significantly to memorization. Interestingly, we also find that an increase in the number of skip connec- tions does not consistently result in higher memorization ratio. For instance, the DDPM++ model with m =3 at spatial resolution of 16x 16 exhibits a lower memorization ratio compared to cases where m or m 2.
Skip connection location. Additionally, we retain a single skip connection but alter its placement within the model architecture. As depicted in Figure 4(b] and Figure 4(d) with the presence of just one skip connection, DDPM+ can achieve memorization ratio exceeding 90%, while NCSN++ attains a memorization ratio surpassing 40%, provided that this single skip connection is positioned at a resolution of 32x This further reinforces that skip connections at higher resolutions play a more substantial role in memorization of diffusion models.
5 Training Procedure T
In Section ? we have demonstrraed that the ratio of memorization increases with the progression of training epochs, indicating the influence of training procedure T on memorization. Consequently, in this section, we delve into the impact of various training hyperparameters on CIFAR-10.
Batch size. In light of prior research highlighting the substantial role of batch size in the performance of discrininative models (Goyal et al. 2017 Keskar et al. 2016), we hypothesize that it may also influence memorization in diffusion models. Therefore, we investigate range of batch sizes, from 128 to 896. Given that we maintain a constant number of training epochs, we apply the linear scaling rule proposed by Goyal et al. (017) to adjust the learning rate accordingly for each batch size. Based on our basic experimental setup, we ensure a consistent ratio of learning rate to batch size x 1011/512 As indicated in Table larger batch size correlates with higher memorization ratio.
Weight decay. Weight decay is typically adopted to prevent neural networks from overfitting, Memorization can be regarded as an overfitting scenario in terms of diffusion models. Motivated by this, we explore its effect on memorization. In our basic experimental setup (see Appendix B we follow Ho et al. (2020); Song et al. (2021b); Karras et al. (2022) to set a zero weight decay. Now we set different values weight decay and show the memorization results in Table leftt.. Consequently, we find that when weight docay rangod between 0 1 10-3 it has a subtle contribution to memorization. When further increasing the value, the memorization ratio drastically decreases.
7

---

Published in Transactions on Machine Learning Research (02/2025)
Table I: Memorization ratios (%) of diffusion models on CIFAR-10 under different batch sizes.
Batch Size
128
256
384
512
640
768
896
Learning rate (10-4)
0.5
1.0
1.5
2.0
2.5
3.0
3.5
=kk
89.52
90.87
91.52
92.09
91.40
92.05
92.77
Rmem (%)
D =2k
56.67
60.36
60.31
60.93
62.61
63.33
61.95
Table 2: Memorization ratios (%) of diffusion models on CIFAR-10 under different (left) weight decay values; (right) EMA values.
Rmem (%)
Rmem (%)
Weight decay
EMA
[D]
lk
[D] =2k
D = 1k
[D] =2k
0
92.09
60.93
0.99929
92.09
60.93
1 x 10-5
91.67
61.63
0.999
91.72
61.38
1 X 10
92.47
61.03
0.99
91.45
59.27
x 10-3
92.11
58.39
0.9
90.16
58.31
1 10
89.07
35.88
0.5
90.42
57.80
x 10
13.79
0.03
0.1
90.78
58.00
1 10-1
1.33
0.00
0.0
90.19
59.20
Exponential model average. EMA was shown to effectively stabilize FIDs (Heusel et al. remove artifacts in generated samples (Song & Ermon 2020). as also shown in Table 6 right). It is widely adopted in current diffusion models (Ho et al. 2020 Dhariwal & Nichol 2021 Karras et al. 2022 Motivated by this, we explore its effect on memorization. Previously, we lix the EMA rate as 0.99929 after the warmup following Karras et al. (2022). As present in Table 2 right), we investigate the memorization of diffusion models with varying EMA rates, from 0.0 to 0.99929. Although EMA values are significant for FIDs, they contribute limitodly to memorization.
2017] and
6 Unconditional v.s. Conditional Generation
It has shown that conditional diffusion models typically yield lower FIDs than their unconditional counterparts (Karras et al., 2022). This observation motivates investigate whether the input condition also exerts an inffuence on the memorization of diffusion modcls. It is worth noting that to train conditional diffusion models, slightly change M by incorporating class embedding layer and adjust T through modifications of the training objective. We show the training objective of dlass-conditionel diffusion models and its optimal solution in Appendix A.3 The experiments are conducted using both EDM (Karras et al., 2022 and Stable Diffusion (Rombach et al. 2022
Class conditioning. As depicted in Figure 5 we note that caas-cooniiionall diffusion models exhibit higher memorization ratios compared to their unconditional counterparts (thus larger EMMs). We hypothesize that this observation is attributed to the additional information introduced by these true class labels. To validate the hypothesis put forth above, we substitute the true class labels with random labels. Intriguingly, we find that the memorization of diffusion models with random labels remains at a similar level to that of models with true labels. This contradicts our previous hypothesis, as random labels are uninformative regarding training images. These results align with earlier research the realm of discriminative models, e-g., Zhang et al. (017)) which suggests that deep neural networks can memorize training data even with randomly assigned labels. It is worth noting that when monitoring the FIDs in Figure 15 we observe that diffussin models conditioned on true labels yield lower FIDs than that random labels. We hypothesize that with true labels conditioning, similar training samples tend to share the same "individual model" for each class, resulting in better "individual models".
Number of classes. When employing random labels as conditions, the class number C is not restricted to 10 in CIFAR-10. Therefore, we manipulate the choices of C' for random labels and then observe their effects on memorization. Initially, we compare Figure P(a) and Figure 5(b) revealing that unconditional diffusion models and conditional models with = maintain similar memorization ratios. However, a discernible trend emerges introduce more classes, with diffusion models exhibiting increased memorization (also observed FFHQ
8

---

Published in Transactions on Machine Learning Research (02/202))
Ratio (%) of Mccaerizaioo
100
Uscond
Ralio (%) Memorizatiioo
100
Ratio (%) Memorizaiion
70-
60
Cood (uzoque)
80
80
Linond
70
Cood (random)
70-
50
60
40
50
40
40-
30
30
5%
20
10
20-
[D] 1k
2k
10:
1k
10
20
50
0.1k
0.5k 1k
2k
45
0k
8%
10k
12%
Data Size |D]
Class Number
Training Epochs
(4) Unconditional conditional
(b) Varying C' for random labels
(c) Unconditional Unique
Figure 5: Memorization ratios (%) of (a) unconditional diffusson models and conditional diffusion models on CIFAR-10 with true / random labels; (b) conditional diffusion models with C random labels;
(c) conditional EDM with unique labels and unconditional EDM at [D] = 50k during the training.
in Figure I())). Intriguingly, with a size of training dataset [D] 5k, even starting at modest 10% memoriza- tion ratio for C =1, conditional diffusion models can attain over 90% memorization ratio with C 1k random labels. Based on these observations, conclude that the nunber of classes significantly influences EMMs.
Unique class condition. We consider an extreme scenario where each training sample in D is assigned a unique class label, which can be regarded as the case of |2| We compare the memorization dynamics of this unique class conditional scenario with true class conditional and unconditional ones during the training, as illustrated in Figure g Notably, within only 8k epochs, the diffusion model with unique labels achieves memorization ratio of more than 95%. On the FFHQ dataset, the memorization ratios of unique class conditional diffusion models are also much higher than that of unconditional ones, as present in Figure 11 (d).. Inspired by this, we extend this condition mechanism to encompass the entire CIFAR-10 dataset, which = contains 1D|=50k: samples. It is worth noting that we train an EDM model Karras et al. (022) to facilitate
a comparison with our initial observations in Section 2 As illustrated in Figure 5(c) the unconditional EDM maintains a memorization ratio of zero throughout the training process even extending it to 12k training epochs. However, upon conditioning on the unique labels, we observe a notable shift on memorixation. The trained conditional EDM achieves more than 65% memorization ratio within 12k training epochs, a significant increase compared to its previous performance. We also visualize the generated images by these two models in Appendix C.3 Lo further validate their memorization behaviors. These results imply that with unique labels, the training samples become strongly associated with their input conditions, rendering them more readily accessible during the generation process when the same conditions are applied.
Extension to text condition. Beyond unconditional diffusion models and class-conditioned diffusion models, we incorporate the experiments on stato-of-the-art text-to-image diffusion models, Stable Diffusion (SD) (Rombach al.] 2022). As it is infeasibly computationally expensive to train Stable Diffusion from ? which is realistic scenario. This dataset consists of C 10 classes from the ImageNet dataset (Deng et al. 2009) and has also been adopted in Somepalli et al. (2023b). We consider two types of text prompts as conditions: (1) plain conditioning: using "a picture" as text prompt. for all images; (2) class condilioning: incorporating class labels in the text prompt, i... "a picture <CLASS NAME>". For fine-tuning, we consider both full fine-tuning and LoRA fine-tuning with different ranks (Hu et al. 2021). More experimental sctups are detailed in Appendix G As shown in Table 3 we compare two distinct lypes of text prompts conditioning under training data with different sizes (D). Firstly, we observe that although SD was pretrained on billions of images, it is still prone to memorize training data when fully fine-tuning it customized data. While LoRA fine-tuning has less tendency to memorize training data. Additionally, we find that with higher LoRA rank, SD models demonstrate higher memorization ratios. Secondly, it is noticed that with the increase of [D], the memorization ion ratio drops. Furthermore, the memorization ratios of SDs fine-tuned on class conditioning data is much higher than that on plain conditioning data, To explain this, class conditioning is similar to the case of a number classes C'= 10 in the dass-conditionee diffusion model while plain conditioning is similar to the case C =1 Our results on Stable Diffusion reaffirm the significant influence of class number on memorization within diffusion models.
scratch, opt to fine-tune the U-Net in SD on the Imagenette dataset
httpps//github.. http:///hhtabcccc///asssssaa/aga
9

---

Published in Transactions on Machine Learning Research (02/2025)
Table 3: Memorization ratio (%) Stable Diffusion (SD) on Imagenette with different fine-tuning
approaches on the Imagenctte dataset under two different text conditioning types.
Fine-tuning
Condition
[D] =400
[D] =500
[D] =80
[D] =1000
[D] -2000
plain
0.00
0.00
0.00
0.00
0.00
LoRA =16
class
0.00
0.00
0.00
0.00
0.00
plain
0.08
0.01
0.00
0.00
0.00
LoRA r 226
class
5.22
0.25
0.00
0.00
0.00
plain
0.76
0.20
0.00
0.00
0.00
LoRA 1024
class
8.77
2.07
0.04
0.00
0.00
plain class
21.95
20.63
7.30
2.85
0.06
Full fine-tuning
46.89
38.79
20.46
10.67
0.21
7 Related Work
Memorization in discriminative models. For discriminative models, memorization and generalization have interleaving connections. [Zhang et al. (2017) first demonstrated that deep learning models memorize the training data, even with random labeling, but generalize well. Additionally, Feldman (220 Feldman &
Zhang (2020) showed that this memorization is necessary for achieving closc-to-optimal generalization under long-tailed assumption of data distribution. In the follow-up works, [Blldokk et al. (2021) |; Stephenson el. al.
(2021) showed that memorization predominately happens in the deep layers while Maini et al. (2023) argued that memorization is confined to a few neurons across various layers of deep neural networks. Although discriminative models can largely memorize training data, this phenomenon does not apply to diffusion models. For instance, [hang et al. (2017) showed that the Inception v3 model (Szegedy et al. under 25M trainable parameters approximately 1.28M images. However, the EDM model (about 56M paramcters) (Karras et al. 2022
2016) with
almost memorize the ImageNet dataset (Deng et al.
2009 with
can not memorize even the CIFAR-10 dataset with 50k images as present in Section 2 From another view, Bartlett et al. (2020); Nakkiran et al.] (2020) showed that over-parameterized models generalize well to real data distribution and even perfectly fit to the training dataset, which is called benign overfitting. Nevertheless, diffusion models demonstrate adverse generalization performance YYon et al. 2023)
Memorization in generative models. (Somepalli et al. (2023a) Carlini et al. 2023 represent the initial explorations on memorization in diffusion models. They investigated range of diffusion models and demonstrated that these models memorize few training samples and replicate them globally or in the object level during the generation. For instance, Carlini et al. (2023) identified only 50 memorized training images from 175 million generated images by Stable Diffusion (Rombach et al.) 2022) and extracted 200-300 training images from 220 generated images by DDPM and its variant {Ho et al., 2020 Nichol & Dhariwal 2021). This highlights the memorization gap between empirical diffusion models and the theoretical optimum defined in equation 2 SSmepalli et al. (2023a b) showed that training data duplication and text conditioning play significant roles in the memorization of diffusion models from empirical studies. Another work (Wen et al.] 2023) introduced novel approach to detect memorized prompts and mitigate memorization for Stable Diffusion (Rombach et al. 2022). However, these conclusions are mostly derived from text-to-image diffusion models. The nature of memorization in diffusion models, especially for unconditional ones, remains unexplored. One recent paper Hintersdorf et al. (2024) localized the memorization to the specific neurons text-to-image diffusion models. Another recent paper Liu et al. (2024) proposed ensembling model parameters and discarding samples with small loss values in diffusion model training, thus mitigating the memorization. Apart from diffusion models, several studies are researching towards memorization in Generative Adversarial Networks (GANs) (Webster et al. 2021 Feng et al. 2021). and language models (Carlini et al. 2021 2022
Kadkhodaie et al. (2023) analyzed the relationship between memorization and generalization. Yoon et al. (2023) hypothesized that generalization of diffusson models is a failure memorize the training data when data size is large and model width is comparatively small memorization-gneerlzzaiooo and this memorization generalization dichotomy may manifest at the level of classes. Our work focuses on the timing of occurrence and the influential factors regarding memorization through much more thorough quanti-
Generalization in diffusion models. Yoon et al. (023):
10

---

Published in Transactions on Machine Learning Research (02/2025)
tative analysis. Kadkhodaie et al. (0233] also showed that diffusion models memorize samples when trained small data. However, it focused on examining the inductive bias of DNN denoisers and showed that generaliza- tion arises from the geometry-adaptive harmonic basis (GAHB) using mathematical formulations. Additionally, Zhang et al. [023) showed that given the same noise input, different diffusion models often generate remarkably similar samples es through a deterministic sampler, manifesting in memorization and generalizat ion regimes. This provides a new perspective on understanding memorization and generalization of diffusion models. One recent paper (Kamb & Ganguli 2024) shares motivations similar to ours. Kamb & Ganguli (2024) identified two in- ductive biases (locality and equivariance) in common architectures of diffusion models that prevent convergence to the ideal score function. Building on this insight, they theoretically derive equivariant local score (ELS) machines, which mirror the formulation of the ideal score function while exhibiting the generalization behavior of empirically trained models. However, these two inductive biases do not fully caplure the behavior of trained models, particularly those with self-attention architectures. Analytical characterization of the exact properties of trained diffusion models remains challenging. Our work, on the empirical side, investigates how data, model designs, training processes, and conditioning prevent trained models from converging to the ideal score function.
8 Conclusion
In this study, we first showed that the theoretical optimum of diffusion models can only replicate training data, representing a memorization behavior. This contracts the typical generalization ability demonstrated by statc-of-the-art diffusion models. To understand the memorization gap, we found that when trained on smaller-sized datasets, learned diffusson models tend approximate the theoretical optimum. Motivated by this, we defined the notion of effective model memorization (EMM) to quantify this memorization behavior. A Afterwards, we explored the impact of critical factors on memorization through the lens of EMM, from the three facets of data distribution, model configuration, and training procedure. We found that data dimension, model size, time embedding, skip connections, and class conditions play significant roles on memorization. Among all illuminating results, the memorization of diffusion models can be triggered by conditioning training data on completely random and uninformative labels. Intriguingly, when incorporating such conditioning design, more than 65% of samples generated by diffusion models trained on the entire 50k CIFAR-10 images are replicas of training data, in contrast to the original 0%. We believe that our study deepens the understanding of memorization in diffusion models and offers clues to theoretical research in the area of generative modeling.
Acknowledgments
We would like to thank our action editor Kamalika Chaudhuri and the reviewers for their detailed and constructive comments and foedback.
References
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems (NeurIPS), 34:10876 10889, 2021.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48)330063 30070, 2020.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Frlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633 2650, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXic preprint XXxi:2202.076466 2022.
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja. Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. arXiv preprint arXiv:2301.1311 2023.
Jia Deng, Wei Dong Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In IEFE Conference on Computer Vision and Pattern Recognation (CVPR), 2009.
11

---

Published in Transactions on Machine Learning Research (02/2025)
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advonces in Neural Information Processing Systems (NewrPPS), 2021.
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advonces in Neural Information Processing Systems (NeurIPS), 33:2881-289) 2020.
Qianli Feng. Chengi Guo, Fabian Benitez-Quirov, and Aleix M Martinez. When do gans replicate? the choice of dataset size. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 6701-6710, 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706. 02677, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6626-6638 2017.
Dominik Hintersdorf, Lukas Struppek, Kristian Kersting, Adam Dziedzic, and Franziska Boenisch. Finding nemo: Localizing neurons responsible for memorization in diffusion models. arXiv preprint ar Xrvv:22006.022 023666 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diflusion probabilistic models. Aduances in Neural Information Processing Systems (NewrIPS), 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint ar arxiv:2106. 09685, 2021.
Rongjee Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. Xix preprint ar arxiv:2301. 12661, 2023.
Zahra Kadkhodaie, Florentin Guth, Ecro Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geomelry-adaptive harmonic representation. arXiv preprinl Xrx::22300.00 0.02557, 2023.
Mason Kamb and Surya Ganguli. An analytic theory of creativity in convolutional diffusion models. arXiv preprint arXiv:24 12 2. 20292, 2024.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE International Conference on Computer Vision (CVPR), 2019.
Tero Karras, Miika Ailtala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems (NeurIPS), 35:26565 26577, 2022.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang, On barge-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint Xrxiv:1600 160.004336 2016.
Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-uss: diffusion model for text-to-speech via classifier guidance. In International Conference on Machine Learning (ICML), pp. 11119-11133. PMLR, 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
Xr 1412.111.000 2.6980, 2014.
Diederik P Kingma and Ruiqi Gao. Understanding the diffusion objective as a weighted integral of elbos. arXiv preprint ar 0rx:2290330000 00848, 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 300-309, 2023.
Xiao Liu, Xiaoliu Guan, Yu Wu, and Jiaxu Miao. Iterative ensemble training with anti-gradient control for mitigating memorization in diffusion models. arXiv preprint urXic:2407.15326 2024.
Pratyush Maini, Michael C Mozer, Hanic Sedghi, Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? arXiv preprint or Xiv:23307 09542, 2023.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations (ICLR), 2020.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Πly Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 10744, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML), pp. 8162 3162-8171. PMLR, 2021.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion.
In International Conference on Learning Representations (ICLR), 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Asse Assisted Intervention
caceaiioooMMC..C (MICCAT), pp. 234-241. Springer, 2015.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilbrriu thermodynamics. In International Conference on Machine Learning (ICML), pp. 2256-2265. PMLR, 2015.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art digital forgery? investigating data replication in diffusion models. In JEEE/CVP Conference Computer Vision and Pattern Recognition (CVPR), pp. 6048-6058, 2023a.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. arXiv preprint arXiv:2305.20086, 2023b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), pp. 11895-11907, 2019.
Yang Song and Stefano Ermon. Improved techniques for training based generative models. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations (ICLR), 2021b.
Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Yeon Chung. On the geometry of generalization and memorization in deep neural networks. arXiv preprint arXi::2105.14400
2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna, Rethinking the inception architecture for computer vision. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818 2826, 2016.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems (NeurIPS), 33: 7537-7547, 2020.
Gerrit van den Burg and Chris Williams. On memorization in probabilistic deep generative models. Advances in Neural Information Processing Systems (NewrIPS), 34:27916-27928, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems (NewrIPS), 30, 2017.
Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusson for graph generation. In International Conference Learning Representations (ICLR), 2022.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compulation, 23 (7):1661-1674, 2011.
Haonan Wang, Qianli Shen, Yao Tong, Yang Zhang, and Kenji Kawaguchi. The stronger the diffusion model, the easier the backdoor: Data poisoning to induce copyright breaches without adjusting finetuning pipeline. arXiv preprint 2rXiv:2401.000 01.00........ 196, 2024.
Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. This person (probably) exists. identity membership attacks against gan generated faces. arXiv preprint arXiv:2107.00018 2021.
Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang, Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations (ICLR), 2022.
Mingyang Yi, Jiacheng Sun, and Zhenguo Li. On the generalization of diffusion model. arXio preprint M.Ki:2305..471.. 2023.
TacHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In 1CML 2023 Workshop on Structured Probabilittic Inference & Generative Modeling, 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR), 2017.
Huijic Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Liyue Shen, and Qing Qu. The emergence of reproducíbility and consistency in diffusion models. arXiv preprint ar Xix:22300. 05264, 2023.
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin. recipe for watermarking diffusion models. Xix preprint or Xim:2333. 10137, 2023.
14

---

Published in Transactions on Machine Learning Research (02/2025)
A Optimal Solution of Diffusion Models
A.1 Derivation of the Theoretical Optimum
In this section, we prove the close form of the optimal score model defined in equation 2 denoising score matching (DSM) objective (Vincent
Firstly, the empirical
2011 is:
N
(DSM((0)
E o. Bem
A(t)
s0 se((eet) +
(5)
2N
01
n=1
1
10,7) A(t)BB
s0 t) 01
(6)
2N
A(t) JO,TT
/
1
Σ
so s0 t)
O(
(7)
2N
oe 26
Compared equation 1 add positive weighting function A(1) > 0, which is normally used in the training of diffusion models (Song et al 2021b). Since the derivative of w.r.t. is de dsc
ax + dic, we have - Therefore,
81
Then
A(t) /
O4 - 2t 2
(D) = Eo,
Σ
s0(2),t)
a of Dordz
(8)
2N
of 2
==1
A(t)) / [DSM 2s, t)dz
= EEn
(9)
To minimize the empirical DSM objective (Ds we can minimize JDSM ze, t),t) given each 2, since X(t) O. The minimization of JDSM (0, t) is a convex optimization problem w.r.t. the score s9(2,, t), which be solved by taking the gradient w.r.l. so(21:)):
0 V (=(())))
1
Σ
so(2),
(10)
2NV
of
N
s(21,))
a, 24
N(21aarnn I
(11)
N Σ
0iD)
80 (0(
N(z 0,1)
04In - 2
(12)
of
n=1
The optimal diffusion model can be written
Em=, N(2 atIn 0,
antn-
8" (z),
(13)
NW1,N(22; of1]
Σ ΣN m== exp
||axn-
207
(14)
Σw= exp
20
N
|01IIn - -2el1
at - 2t
(15)
20
#=1
where S refers to the softmax operation.
It is noted that when s0(2,,t) t) is parameterized using neural network 8, the objective equation typically highly non-convex w.r.t. 0. Therefore, achieving this theoretical optimum necessitates model 0
15

---

Published in Transactions on Machine Learning Research (02/2025)
with sufficient capacity. Firstly, this theoretical optimum may lie outside the solution space of 0's model family. Even if the solution space encompasses the theoretical optimum, reaching it remains challenging due to the non-convex nature of the optimization in equation 5 w.r.t. model parameters 0. The optimization may get stuck in the local optimum.
Given the existence of theoretical optimum, we find that the empirical DSM objective can be rewritten
N
(0) = 2N
Σ n=1
E (A())
[| + tzc,tt - s (a,TT + oee, 1)11 + +C,
(16)
where - E. XO 8* (z(; -000)))--00) N Dz a constant value without
involvement of the trained diffusion model 0. This equivalence shows that diffusion models are trained to approximate the theoretical optimum. The proof is shown as below
(O)
(17)
=E,
X(t)
1
Σ
()(
04
(18)
2N
o²
/
N
EE
X(t)
Σ
s0(21, 1) (21, + s (21,t)
0u - 2t
o
(19)
2N
==
E,
X(1)
||50(2e, () - -22 ((, ar Nodde
(20)
2N
#=
E,
1 X(t)
Σ
2 (so (26, =s2(2t, t))
8 (2,,
Ogtin - 2t
N of
2N
of
+EE
X(t)
1
Σ
(2(,t)
atIn 24
2.V
o?
A(t) /
E,
[50(2e,)) 1) - sx s" (22 a. of No
+C
(21)
2N
==1
N
E1,
[A(() (0sTa + t,e,tt - s" (01zn + oic, +C.
(22)
2N
==1
The above equation holds since
ar =0) Σ 2 (20(2)) (z,,t) 8" (z(,))
N Nazroronn oi ofl)o
(23)
2N
oā
N
N (so(2,,t) t) - -s- (zn.t) / Σ NZz
(2((2t))
attn - 24 dzs
(24)
(25)
In Karras et al. 2022), the authors have derived the optimal denoised function, while in Yi et al. 2023 the authors provided the closed form of the optimal DDPM (Ho et al. 2020 Therefore, we show the equivalence of our equation 2 with the above two forms.
For DDPM, which is trained under noise prediction objective, between the two parameterizations:
Kingma & Gao (0223) gave the transformation
60(21,
so(2,,
(26)
16

---

Published in Transactions on Machine Learning Research (02/2025)
Therefore, the optimal DDPM can be represented
Σn Ma 2t ot
c² (21,)) -2,8"
(27)
Ex= ((z 0211
||a, 109n-1zll 2
exp
20
In
(28)
nm
2 exp
20;
In DDPM (Ho al. 2020), the forward process is defined as 2t = va + vI CcC, so the closed form for optimal DDPM is
|| 1 In
exp
2(1-(t)
(=t, VT- at
(29)
VI C1
21 -, 12 2(1-as)
n/==1 cxp
which is the same as Theorem 2 in et al. 2023
For denoised function, which trained under data prediction objective,
Kingma & Gao
2023
gave:
D,(o so s0(2),
(30)
of
Therefore, the optimal denoised function can be represented as
D' (2,, t) 2t
n= M(z of of
(31)
Ex (2:
EDM (Karras et al. 2022). the forward process is defined as 2 = x oze, so the closed form for optimal denoised function is:
t²,, Em N22 o
(32)
Ew= (2ziow o
which is the same as Eq. (57) in Karras et al. (2022).
A.2 Backward Process of the Optimal Diffusion Model
We analyze the memorization behavior of the optimal diffusion model s²(,,, defined in Eq. equation through the lens of backward process. As shown in Kingma et al. (2021). the backward process of our defined diffusion models is governed by the following stochastic differential equation (SDE):
dæ, = [f(t) (t))ss((2),)) t)) dt + gt)ddW,
(33)
where We a standard Brownian motion, and ((t and g(l) follow
f(t) d'log a dt
(²) do? dllogar ar of
(34)
dt
Besides the SDE, Song al. (2021b) showed that there exists an ordinary differential equation (ODE) for deterministic backward process
[00) g (1))(())))) dt.
1
dz,
(35)
We first show how adopt the above ODE to generate samples using the optimal score model defined in equation 2 Specifically, sample multiple time steps 0 to < < l in T, where & refers to
17

---

Published in Transactions on Machine Learning Research (02/2025)
small value close to 0 and T > 0 represents the maximum time step. For simplicity, consider Euler solver, and then we have the following update rule
(dlogar ai
do
dlog at of so(z)1))
2a 2ast
+
dl
22
2 di
2
dl
(In - bxt1).
(36)
We also use Euler method to approximate
log as and dt
de
considering
dt
dllog ay dt
1 dou
Ot atat]
lim
lim
lim
(37)
ba 1141
In bn-lnt-
Or dt
In-n++10
Oea+1 tn bnt1
do?
20, do
Tt. 0t
lim
lim
lim
20t in bn+1
(38)
dt
ta ta-tn41-0
dt
1= tu fn-ta41-00
Then we have
(as, atn+ )o spl (p(2 bnt1)
at Mm--
2t.
Otat (Orm - otots)
(39)
0ta+1
at o2 Tnll
(t Ztn-1
Otat Otn
80 (Stmer bntt)-
(40)
06411
ah++1
For to 0, we know that ag and do 0. 20 refers the generated samples, and we have
o²
2 +
20
(2(1)
(41)
ag
|10e -eell
2
exp
m8 +
20
Or
-g
(42)
ag
ag
N
2n'=1 exp
|10.0t,, 20
o
exp
20
(43)
||ac =2
m/=1 exp
20
From the above equation, conclude that the generated samples by the optimal diffusion model are the linear combinations of training samples in D.
Next, we consider a discrete distribution, and suppose 2
then aq
&c
1|=-2 2020?
exp
p(z xn.
1,2,...NN
(44)
Σ exp
1|=- 11
2
Suppose = N2, D) aig ||2 x:13. then we have
|| Jm - xn < 0.
(45)
18

---

Published in Transactions on Machine Learning Research (02/2025)
When & ++ 0+, 17 açōş - 0+, then we have
||=- 11
exp
211²
lim p(= Tm. {) = lim f-10-
(46)
n-+0
2n'=1 exp
.N
1|2 1
2n1*
lim 7-00
(47)
2nlzm exp (-
|| 113
1| -m Zm
+
211
(48)
En' lim exp
118
11.
1/ 1 11
+
2n²
1
(49)
En' liml exp
[1 Im 112 - |1 Jxne 11:))
1.
(50)
Then
p(z Im 5) + Σ p(z Zee x3)
lim {-0+
(51)
Σ
lim p(2 () O. 8
(52)
12' n'
Given p(= = x4, 2 0,
lim p(= = Drgsim {) = O. E-+++
(53)
lim lim Bp(==un.5) [=]
(54)
<
<-+0+
From the above analysis, conclude that when t = & is closed to 0, the probabilistic ODE solver returns training sample in D.
Next, we consider an SDE solver, then the update rule is
ddlogar
dož
dlog aoggooo of so(zut))
24 2nti +
21
2
(tn tn+1)
(55)
dt
dt
dt
l=but
do?
dloggr an
+ dl
2
0
(tn tnti)e
dt
1=h111
ata oz
at. Ztn
or, Ot,
(56)
OH,,11
att+11
Ot, Pnn1
+ 2 Ota-1
+V
tre
(tn - tntile,
att+1
where e I), is a Gaussian noise, Similarly, we consider the update step at to 0
20
+ 2 so (2c2 E + 12
ofc
c=
2
Σ
N
exp
2ng
28
22
c.
(57)
ag
ag
age
n=1
N 2n'=1
exp
20
11
ag
ag
When & + 0+
E9 D
Zg
lim 20 2NN,
lim
(58)
80
0
.
0g
19

---

Published in Transactions on Machine Learning Research (02/2025)
100
12
Ratio (%) of Memooizatiin
95
D lk D 2k
Ratio (%) Mcmorization
10
90
85
8
80
DDI
75
6
D 2k
70
65
60
2
55
50
5k
10k
15k 20k 25k 30k 35k 40k 45k 50k
5k
10k
15k 20k 25k 30k 35k 40k 45k 50k
Sample Size
Sample Size
Figure 6: (Left) Memorization ratios (%) (Right) Standard deviations of memorization ratios (%) of diffusion
models with different sample sizes.
Consider lime-+0 2t -imm-00 20
lim f-01
20
NN,
2 D
(59)
ag
To summarize, through our analysis, we find that the optimal diffusion model always replicates training data through the backward process.
A.3 Optimal Cladstcond Diffusion Model
In the above, we provide the derivation of the optimal diffusion model for unconditional generation under the assumption of empirical data distribution ~ * E 8(x - In). Next, we consider the scenario of class- conditional generation. The dataset can be represented as D {In, Un 1 Un E (1,2,.... C), where C is the number of classes. Then the empirical joint distribution of data and label is x, y N Σn Ō(w 2n) ya).
For class-conditional generation, the empirical DSM objective is written
(D) = 2NV
Er,
A(t)
se (arxn + t,e, Vn.
6
(60)
0:
n=1
N
E,
B
A(t)
salayz t,,,ctt)
2N
0
where za refers to the n-th sample with class label and No represents the number of class label Similarly, by taking the gradient w.r.t. s0(2); G, t), we derive the optimal class-conditioned diffusion model for each class condition
Exe exp (- Iare -2el12)
(-laras-
aiz
No. Σ S
2, 113
207
o?
a,2 2t
(2 c,t)
(61)
207
of
No exp
n=1
20
B Implementation Details of Our Basic Experimental setup
Data distribution. Most of our experiments are conducted on the CIFAR-10 dataset, which consists of 50k RGB images with a spatial resolution of 32x32. CIFAR-10 has 10 classes, cach of which has 5k images. We also incorporate the experiments on the FFHQ dataset (Karras al., 2019] emphasize our findings. The FFHQ dataset comprises 70k RGB images of human faces with original resolution of
20

---

Published in Transactions on Machine Learning Research (02/2025)
1024x1024. We down-sample these images to resolution of 64x64 following Karras al (2022) to conduct our experiments. When modifying the intra-diversity of data distribution, we blend several images from the ImageNet dataset (Deng et al. 2009) to construct training datasets. In our study, we disable the data augmentation to prevent any ambiguity regarding memorization.
Model configuration. We consider the baseline VP configuration in Karras et al. 2022 3 The model
architecture is DDPM++, which is based on U-Net (Ronneberger et al. 2015). As our basic model, we select the number of residual blocks per resolution in the U-Net as 2 instead of 4 in original implementations. For diffusion models trained 11FAR-10, the channel multiplier is 128, resulting in 256 channels at all resolutions of 32x32, 16x16, and 8x8. While for diffusion models trained on FFHQ, the employed U-Net has 128 channels at the resolution of 64x64, and 256 channels at the resolutions of 32x32, 16x16, 16, and 8x8. The time embedding is positional encoding |Vaswani et al. 2017).
Training procedure. Our diffusion models are trainod using Adam optimizer (Kingma Ba 2014) with a learning rate of 2 10-4 and a batch size of 512 for CIFAR-10 while 256 for FFHQ. The training duration is 40k epochs, while it is 4k epochs in Karras et al. (0022). It is worth mentioning that for different training sizes, the number of training epochs is the same. Therefore, for smaller training datasets, the number of total training steps will be smaller. This setup ensures that the frequency of each image being drawn during the training procedure is the same. We schedule the learning rate and EMA rate similar to Karras et al. (2022) but in an epoch-wise wise manner. In the first 200 epochs, we warm-up the learning rate and the EMA rate with the increase of training iterations. Afterwards, the learning rate is fixed to 2 x 10 and the EMA rate is fixed to 0.99929.
Gencration and evaluation. We follow the backward process in Karras et al. 2022 to generate images to compute the memorization metric. To decide appropriate sample size, we first train two diffusson models on CIFAR-10 according to our basic experimental setup at [D] 1K and [D] = =2k. Afterwards, we generate 50k images by each model and then bootstrap different number of samples to compute memorization ratios. As present in Figure G we find that the ratio of memorization has a negligible variance when sample size more than 10k images. Therefore, we generate 10k images throughout this study. We report the highest, memorization ratio during the training process. To ensure the consistency in our analysis, we maintain the same set of 10k samples for computing FIDs and alternative memorization metrics. Specifically, we compute FIDs using 10k generated samples and the training datasel D (with varying sizes (D]).
Experimental compute resource. All experiments were run on 8 NVIDIA A100 GPUs, each with S0GB wapeheepppeoeee
of memory. The running time of each experiment highly depends on the model configuration and data size. For example, it lakes about 120 hours to train an unconditional model or conditional model with unique labels in Figure 5(c)
Experimental statistical significance. We also evaluate the variance of memorization ratios across multiple trials. Specifically, on CIFAR-10, we run a pair of experiments focused on data distribution, primarily by altering the number of classes. We set the training data size as |D] 1k and the mumber of classes as 2 5, and then execute each experiment over three distinct random seeds. Cousequently, the memorization ratio for C 2 stands at 94.59=0.19%, whereas for C' 5, 5, it is recorded at 92.32+0.144% It is noticed that the variance of memorization ratios is less than 0.2%, which is insignificant. Considering this minimal fluctuation and difficulty for repeating all exporiments, we run cach experiment once in the main paper.
C More Empirical Results on CIFAR-10
C.1 Model Size
Model width. We investigate the influence of different channel multipliers, specifically exploring the values from (64, 96, 128, 160, 192 224, 256). We also consider two scenarios for the nunber of residual blocks per resolution: 2 and 4. As illustrated in Figure 7(a) when [D] = 2k, we observe a consistent and monotonic increase in the memorization ratio as the model width grows. This observation aligns with the conclusions
We run the training con foniuuration Karras et al. 2022 after adjusting hyper- parametrsss and redistributing capacity compared Song al. (221bb).
21

---

Published in Transactions on Machine Learning Research (02/2025)
drawn in Sec.
4.1 Furthermore, we provide dynamic view of the memorization ratios during the training process in Figure 7(c) and Figure [ Tt is worth noting that wider diffusion models consistently exhibit.
higher levels of memorization throughout the entire training process.
Model depth. We delve into the impact of varying numbers of residual blocks per resolution, considering values in the range of (2, 4, 6, 8, 10]. We maintain two different channel multiplier values, specifically 128 and 256, and set the training data size [D] 22. The results, as presented in Figure [[b] confirm the non-monotonic effect of model depth on memorization. When the channel multiplier is set to 256, the curve depicting the relationship between memorization ratio and the number of residual blocks per resolution exhibits multiple peaks. To gain a better understanding of this non-monotonic effect, we visualize the training process in Figure 7(d) and Figure 7(1) Occasionally, deeper diffusion models yield lower memorization ratios than shallower ones throughout the whole training process. It is noteworthy that when both model width and depth are set large values (e-g., the number of residual blocks per resolution is and the channel multiplier is
256), the memorization of the diffusion model also demonstrates non-monotonic characteristics.
C.2 Skip Connections
We employ DDPM++ (Song al. 2021b: Karras et al., 2022 to explore the influence of skip connection
quantity on memorization. Specifically, we consider retaining 1, 3, 5, 7, 8, skip connections. To keep the costs tractable, we randomly select five distinct combinations of skip connections for each specific m. Our findings, depicted in Figure [(a] illustrate a consistent memorization ratio for larger values of m, whereas a considerable degree of variance is observed for smaller values of m. Notably, when m =1 the memorization ratio exhibits substantial variability, spanning a range from approximately 0% to over 90%.
In addition remaining spocific skip connections at different resolutions in the main paper, we also investigate the effect of selectively deleting certain skip connections varying resolutions. As present in Figure [(b] when deleting one two skip connections any spatial resolution, the memorization ratios consistently remain above 90%. It is noteworthy that the memorization ratio only falls below 90% when three skip connections are deleted at the resolution of 32 These findings further confirm the pivotal role played by skip connections at higher resolutions on memorization of diffusion models.
C.3 Unconditional v.s. Conditional Generation
We compare the memorization ratios of unconditional diffiusion models, conditional diffusion models with true labels, and conditional diffusion models with unique labels. As summarized in Table 4 it is noticed that with unique labels as class conditions, the trained diffusion models can only replicate training data as the memorization ratios achieve close to 100% when |D] 5k. Additionally, we visualize the memorization ratios of diffusion models during the training in Figure g which reaffirms that the diffusion models with unique labels input conditions memorize training data quickly. Typically, they can achieve more than 90% memorization ratios within 8k training epochs.
Finally, when trained on the entire CIFAR-10 dataset, i.c., [D] = =50k, the memorization ratio of conditional EDM with unique labels achieves more than 65% within 12k training epochs, as present in Figure 5(c) However, its unconditional counterpart still maintains a zero value of memorization ratio. To further demonstrate this memorization gap, we visualize the generated images and their nearsstt training samples in D by the above two models in Figure [0] It is noticed that the conditional EDM with unique labels replicate large proportion of training data while the unconditional EDM generates novel samples.
D More Empirical Results on FFHQ
D.1 Data Dimension
We investigate the influence of data dimension on the memorization of diffusson models, particularly those trainod using the FFHQ dataset. Specifically, we evaluate various resolutions: 64x64, 32x32, 16x16, with the latter two resolutions achieved by downsampling. We keep the model configurations and training procedure the same. As shown in Figure [1(a) for the 64x64 input resolution, observe EMM between 500 and
22

---

Published in Transactions on Machine Learning Research (02/2025)
100
100
Ratio (%) of Memorizztiion
Ratio (%) of Mcmorization
90
95
80
70
85
60
80
50
75
40
30-
20
Depth
65
Wadth 256
10
Depth
60
Width
0
64
96
128
160
192
224
256
2
10
Channel Multiplier
Block Number
(b)
100
Ratio (%) of Memorizatioo
100
Ratio (%) of Memorization
90
90
80
70
70
60
60
50
Width
Depth = 10x
40
Width = 224X
40
Depth = 8x
30
Width = 192x
30
Depth 6x
20
Width 160X
Depth 4x
10
Width = 128X
10
Depth = 2x
0
8k
16k
24k
32k
40k
8k
16k
24k
32k
40k
Training Epochs
Training Epochs
(d)
Ratio (%) of Memorization
100
Ratio (%) Memorization
100
90
90
80
70
70
60
60
50
Width
Depth = 10x
40
Width = 224x
40
Depth = 8x
30
Width = 192x
30
Depth 6x
20
Width 160X
Depth = 4x
10
Width = 128X
10
Depth = 2x
0
8k
16k
24k
32k
40k
8k
16k
24k
32k
40k
Training Epochs
Training Epochs
(f)
Figure 7: At the size of training data |D] =2,, memorization ratio (%) of diffusion models on CIFAR-10 with (a) varying model widths under 2 residual blocks per resolution; (b) varying model depths under the channel multiplier as 128 or 256; (c) varying model widths under 2 residual blocks per resolution during the training; (d) varying model depths under the channel multiplier as 128 during the training: (c) varying model widths under residual blocks per resolution during the training; (f) varying model depths under the channel multiplier as 256 during the training.
1k. While for the 32x32 input resolution, the EMM close to 4k. Furthermore, when input resolution is 16, the memorization ratio still above 90% even for [D] = 8k. The EMM is estimated to be between and 10k. These results further indicate the significance of data dimensionality on the memorization within diffusion models.
23

---

Published in Transactions on Machine Learning Research (02/2025)
100
100
Ratio (%) of MMmooizaziinn
90
Ratio (%) oemoriatiion
80
80
70-
70
60-
50-
50
40
40
30-
Skip Num 3
20
20
Skip Num 2
10-
10
Skip Num =
0-
A
3
7
8
9
32x32
16x 16
8X8
Skip Num
Spatial Resolution
(a)
b
Figure 8: Memorization ratio (%) of diffusion models on CIFAR-10 when (a) retaining different numbers
of skip connections; (b) deleting skip connections of certain spatial resolution.
100
100
Ratio (%) of Memorizatiin
90
Ratio (%) Memorization
90
Cond (unique)
80
80
Cond (truc)
70
Uncond
60
60
50
50
40
40
30
Cond (unique)
20
Cond (truc)
20
10
Uncond
10
8k
16k
24k
32k
40k
8k
16k
24k
32k
40k
Training Epochs
Training Epochs
(a)
(b)
100
100
Ratio (%) of Memorization
Ratio (%) Memorization
90
90
80
80
70
60
Cond (unique)
60
Cond (unique)
50
Cond (truc)
50
Cond (truc) Uncond
40
Uncond
40
30
20
20
10
10
8k
16k
24k
32k
40k
8k
16k
24k
32k
40k
Training Epochs
Training Epochs
(c)
(d)
Figure 9: Memorization ratios (%) of unconditional / conditional (with true labels) / conditional (with unique labels) diffusion models on CIFAR-10 during the training at the training data size (a) |D] = 2k; (b)
(D) = 3k: (c) [D] = 4k; (d) 5k.
D.2 Time Embedding
We conduct experiments compare two distinct time embodding methods within the model structure of
DDPM+ (Song et al. 2021bb Karras al. 2022);: positional embeeding (Vaswani et al.) (017)) and random
fourier features (Tancik et al. 2020 As illustrated in Figure 11(b] by varying training data size [D], there
24

---

Published in Transactions on Machine Learning Research (02/2025)
Table Memorization ratios (%) of unconditional diffusion models on CIFAR-10.
conditional (true labels) / conditional (unique labels)
[D] = 1k
[D]
2k
[D] = 3k
[D]
4k
[D] = 5k
Unconditional Conditional (true labels)
92.09
60.93
34.60
17.12
9.00
96.59
79.46
55.62
37.59
23.00
Conditional (unique labels)
100.00
99.88
99.89
99.57
99.66
(b)
Figure 10: Generated images (lop three rows) and their fz-nearest training samples in D (bottom three rows) by (a) the conditional EDM with unique labels (b) the unconditional EDM.
is notable decline in the memorization ratio when employing random fourier features for time embeddings in the DDPM++ model, which reconfirms our conclusions in the main paper.
D.3 Unconditional v.s. Conditional Generation
In the main paper, we demonstrated the significant contributions of random labels to memorization within diffusion models. To further substantiate these findings, we conduct additional experiments using the FFHQ dataset. We note that FFHQ has no ground truth class labels for each image, so only consider random labels and unique labels in our experimental design. Firstly, Subsequently, the number of classes C, for random labels, is varied within the set {1, 10, 50, 100, 200, 400, 2000}. Afterwards, we train conditional diffusion models on these training data of different random labels. As demonstrated Figure [1(c) the memorization ratio for C 0=1 (equivalent to the unconditional scenario) approximates to only 20%. However, it achieves over 90% with an increase in C to 400. Additionally, we conduct another experiment to investigate the effects unique labels. As visualized in Figure [d] when unique labels are provided
construct training dataset with [D] = 2k.
conditions to diffusion models, the memorization ratio still exceeds 90% at training data size of [D] = -kk. In contrast, for unconditional diffusion models, the memorization ratio diminishes even below 90% when |D] = 1k. To summarize, these findings show that the number of classes for random labels plays pivotal role in the memorization of diffusion models, thus aligning with our earlier results derived from experiments on the CIFAR-10 dataset.
E Experimental Results of FIDs
In this section, we provide the Fréchet Inception Distances (FIDs) Heusel 2017) in parallel with the memorization ratios in the main paper. The results related to CIFAR-10 shown in Figure (12 15 and
25

---

Published in Transactions on Machine Learning Research (02/2025)
Ratio (%) Memoriztion
100
100
90
Ratio (%) of Memorization
90
FourRt
Positional
80
70
70
64X64
60
60
50
32x32
40
16x16
40
30
30
20
10
10
0.5k
lk
2k
4k
8k
12k
0.5k
Data Size
|D|
Data Size |D|
(a)
(b)
Ratio (%) of Memorization
100
Ratio (%) of Mcmorization
100
90
|D|
90
80
70
70
60
60
50
40
40
30
30
20
20
Uncond
10
10
Cond (unique)
10
50 100200400
2000
0.5k
1k
2k
Class Number
Data Size [D]
(d)
Figure Memorization ratio (%) of diffusion models on FFHQ under (a) different data dimension; (b) different time embeddings; (c) different numbers of classes for random labels; (d) unconditional diffusion models and conditional diffusion models with unique conditions.
Ras 32X32
16
C-
1.0
Ras 16X16
14
C=5
0.8
C-2
24. 0.5
12
21
RD
10
18
0,0
12
8
0.5k
1k
2k
4k
8k
16k
0.5k
1k
2k
0.5k
1k
2k
Data Size [D0]
Data Size [D]
Data Size [D]
(a)
(b)
(c)
Figure 12: FIDs of diffusion models on CIFAR-10 under different (a) data dimensions; (b) inter-diversity; (c) intra-diversity.
Table 6 While the results FFHQ present in Figure 16 By analyzing the FID results, observe that configuration with higher memorization ratio tends to results in lower FID. This observation is in line with the intuition: when diffusion models memorize a significant proportion of training data, FIDs are also close to optimal.
26

---

Published in Transactions on Machine Learning Research (02/2025)
16
18
80
2x
16
Dtpli oppliwcx
Dipt: WX
Topé: "I
Poator, disk 2
Dyp
Portin, Light
10
Fostbeal Aept m2x
Fostinal
20
10
0.3k 1k
3k Data Size [Z]
4k
3k
0.5k
1.0k
1.3k
2.0k
2.5k
5.0k
0 0.3k
2x
Dana Size
Data Size [D]
(a)
(b)
(c)
Figure 13: FIDs of diffusson models on CIFAR-10 under different (a) model widths; (b) model depths; (c) time embeddings.
60
Ricm 10015
70-
50-
40-
Step
10-
t0x15
Spetial Beseztion
Lacex Skip Connecionn
Spuzal Roookziix
Indes of Skip Connactinn
(a)
(b)
(c)
(d)
Figure 14: FIDs of diffusion models on CIFAR-10 when retaining (a) skip connections of certain spatial resolution for DDPM++; (b) single skip connection at different locations for DDPM++; (c) skip connections of certain spatial resolution for NCSN++; (d) single skip connection at different locations for NCSN+
Table 5: FIDs of diffusion models on CIFAR-10 under different batch sizes.
Batch Size
128
256
384
512
640
768
896
Learning rate (10 (10-4)
0.5
1.0
1.5
2.0
2.5
3.0
3.5
D = =1k
8.89
8.60
8.14
7.99
8.25
7.98
7.85
R Rmem (%)
D =2k
17.56
15.57
15.23
14.41
14.83
14.56
14.85
Table 6: FIDs of diffusion models CIFAR-10 under different (left) weight decay values; (right) EMA
values.
FID
FID
Weight decay
EMA
[D] = 1k
[D] -2k
[ =1k
|D] =2k
0
7.99
14.41
0.99929
7.99
14.41
1 10-5
8.17
15.73
0.999
8.11
15.75
1 1
8.08
14.36
0.99
8.55
17.11
1 X 10-3
9.37
12.75
0.9
9.83
18.04
1 X 10-2
13.47
17.02
0.5
10.32
21.28
5 X 10-2
41.08
51.70
0.1
12.38
23.12
1 10-1
63.78
48.21
0.0
11.77
22.39
27

---

Published in Transactions on Machine Learning Research (02/2025)
43
43
Uncond
39
D 5k
33
Cond (random)
35
D
2k
28
Cond (true)
31
D
Cond (unique)
27
23
23
18
19
15
13
8
7
3
1k
2k
3k
4k
5k
3
20
50 0.1k
0.5k 1k
Data Size |D]
Class Number
(b)
Figure 15: FIDs of (a) unconditional diffuuiio models and conditional diffusion models with true / random unique labels on CIFAR-10; (b) conditional diffusion models with random labels of different C.
90
24
Res = 64x64
80
21
Res = 32x32
70
Res = 16x16 16
60
15
50
Fourier
12
40
Positional
30
6-
20
3
10
0.5k
1k
2k
4k
8k
12k
0.5k
1k
2k
Data Size |T]
Data Size DD|
(a)
(b)
27
24
|D| = 2k
24
Uncond
21
21
Cond (unique)
18
15
12
12
9
3
0:
50 100 400400
2000
0.5k 1k
2k
3k
4k
5k
Class Number (c)
Data Size |D]
(d)
Figure 16: FIDs of diffusion models on FFHQ under (a) different data dimension: (b) different time embeddings; (c) different numbers of classes for random labels; (d) unconditional diffusion models and conditional diffusion models with unique conditions.
28

---

Published in Transactions on Machine Learning Research (02/2025)
18
KNN Distance
Ris 32X32
KNN Distance
14
16
Distance
a= 1.0
20
5X8
C=2
0.5
16
10
C=1
15
0.2
12
a 0.0
0.5k
2k
4k
8x
16k
0.5k
0.3k
1k
Size
[D]
[D]
Size [D]
(a)
(c)
Figure 17: KNN distances of diffusion models CIFAR-10 under different (a) data dimensions; (b) inter-diversity; (c) intra-diversity.
16
Wiz2 X21X
Pyp
Feaker deplo IX Agtz
Wint
XNN Disanee
12
Widt 3301
Wast 2514
KNN Distane
12
10
KNN Disannee
21
0.5k 1k
2k
34
56
0.3%
1.0k
1.5k
2.0k
2.5k
5.00
0.3
2x
Data Size [[]]
Data Size [D]
Data Size [D]
(a)
(c)
Figure 18: KNN distances of diffusion models CIFAR-10 under different (a) model widths; (b) model
depths; (c) time embeddings.
F
Experimental Results of Alternative Memorization Criteria
In this section, we aim to corroborate our findings regarding memorization of diffusion models by incorporating alternative memorization metrics. The memorization ratio, as defined in the main paper, can be formally articulated follows
N'
I NN'. D)ll2 1 < 1. NN2(1', D)112 })
RMem
II(
(62)
R=1
where I an indicator function, NN D) the j-th nearest training sample of x/ D, and N' the number of generations sampled by the diffusion model.
We also consider the KNN distance used in Carlini et al. 2023
surrogate memorization metrie:
N
M
N7 Σ 1/ (N D)
(63)
It is noticed that when memorization ratio is high or KNN distance is low, the diffusion models feature memorizing more on the training data. Afterwards, we re-evaluate our results in the main paper by using the KNN distance as memorization metric. The results on CIFAR-10 are shown in Figure [17 and Table 7
while the results related to FFHQ are present in Figure 21 We notice that these new results are in alignment with our original conclusions using the memorization ratio metric.
29

---

Published in Transactions on Machine Learning Research (02/2025)
Rice-320 32x02
54
KNNN
24
24
15x15 Spacil Resobrion
TERIC Sperial Rosolution
Caten of Commecioo
Iedex Skip Connectior
(a)
(b)
(c)
(d)
Figure 19: KNN distances of diffusion models on CIFAR-10 when retaining (a) skip connections of certain spatial resolution for DDPM++; (b) single skip connection al different locations for DDPM++: (c) skip connections of certain spatial resolution for NCSN++; (d) single skip connection at different locations for
NCSN+
30
27-
27
[2] 5k
KNN Distance
24
21
28
18
Unoond
12
9
ANN
Cand (anique)
Cond (candian)
5k
10
50 0.1k
0.5%
4k
6%
Sk
10k
12k
Data Size [D]]
Training Epochs
(a)
(b)
(c)
Figure 20: KNN distances of (a) unconditional diffusion models and conditional diffusion models with true labels or random labels on CIFAR-10; (b) conditional diffusion models with random labels of different C; (c) conditional EDM with unique labels and unconditional EDM [D] 50k during the training.
Table 7: KNN distances of diffusion models on CIFAR-10 under different batch sizes.
Batch Size
128
256
384
512
640
896
Learning rate (10-4)
0.5
1.0
1.5
2.0
2.5
3.0
3.5
D = 1k
7.35
6.73
6.51
6.35
6.51
6.40
6.30
KNN distance
D 22
15.43
14.92
14.88
14.77
14.53
14.12
14.57
Table 8: KNN distances of diffusion models on CIFAR-10 under different (left) weight decay values; (right)
EMA values.
KNN distance
KNN distance
Weight decay
EMA
[D] =1k
[D] =2k
D =1k
[D] =2k
0
6.35
14.77
0.99929
6.35
14.77
1 x 10
6.45
14.63
0.999
6.44
14.58
1 -
6.42
15.29
0.99
6.51
15.21
1 10
7.32
16.99
0.9
7.02
15.39
1x 10-2
9.99
23.40
0.5
7.11
15.04
5 1
26.69
38.73
0.1
6.90
15.08
1 -0
34.44
39.51
0.0
7.08
14.59
30

---

Published in Transactions on Machine Learning Research (02/2025)
32
40
28
36
Fourier
32
Postiical
KNN Distance
24
20
Res 64x64
24
16
Res 32x32
20
12
Res = 16x16
2
16
12
8-
0.5k
1k
2k
4k
8k
12k
0.5k
1k
2k
Data Size [D]
Data Size |D]
(a)
(b)
30
30
27
|a| = 2k
27
Uncond
24
24
Cond (unique)
KNN Distance
KNN Distance
21
18
15
15
12
12
9
6
10 50 200400000
2000
0.5k
1k
2k
3k
4k
5k
Class Number
Data Size [D]
(c)
(d)
Figure 21: KNN distances of diffusion models on FFHQ under (a) diffferent data dimension; (b) different.
time embeddings; (c) different numbers classes for random labels; (d) unconditional diffusion models and conditional diffusion models with unique conditions.
G Implementation Details of Finetuning Stable Diffusion
Data distribution. Imagenette consists of C 10 classes from the ImageNet dataset (Deng et al. 2009 The classes include "chain saw", "garbage truck", "tench", "French horn", "gas pump", "English springer". "parachute", "church", "cassette player", "golf ball". In our experiments, the image resolution is specified as 256x256. Similar to our experiments on CIFAR-10 and FFHQ, we disable all data augmentation.
Model configuration. Stable diffusion (Rombach et al. 2022 includes an image encoder, an image decoder, a unet, and a text encoder. The image encoder is used to encode images into latent representations while the decoder reconstructs images from the latent. The unet plays role as the latent diffusion model conditioning text embeddings. Before fine-tuning the model the Artbench dalaset, load the model weights from pre-trained stable diffusior
Training procedure. During the fine-tuning, we only train the unet in stable diffusion and freeze the model weights image encoder /deeer/eeoderr and text encoder. We adopt a learning rate of 1 x 10-4 and 3 x 10-1 for full fine-tuning and LoRA fine-tuning (Hu et al. 2021 with cosine learning rate schedule and weight decay of x 60-6, The batch size is set as and the gradient accumulation step is set as 4. For each training dataset. D, we enable the EMA and fine-tune the stable diffusion until 400k images have been observed.
Text-to-image generation. We follow the pipcline of DDIM (Song al. 2021a) with 50 steps to sample 10k images from fine-tuned stable diffusion. During the generation, we disable the safety checker to prevent generating black images. For plain conditioning, we use the text prompt "a picture".. While for class conditioning, we sample 1k images using the prompt "a picture <CLASS NAME>" NAME>> for each class, totaling 10k generated samples.
31

---

Published in Transactions on Machine Learning Research (02/2025)
Ratio (%) Memorizatio
90
0.05
Raio Memoriaatiss
100
80
90
80
70
80
50
40
30
52X32
70
Width 125
|2]
20
Res KX
Res 16X16
60
Widhh 235
Widh 1234
Widdh 1924
20-
[D] kk
[2]
0.5k
1k
2k
4k
8k
16k
50 0.3k
4k
50 0.1k
0.3k
Data Size [D]
Data Size ((]
Class Number
(a)
(b)
(c)
Figure 22: EMMs of diffuuson models with various under configurations of (a) different data dimensions; (b) different model widths; (c) different numbers of classes for random labcls.
H Sensitivity of Effective Model Memorization on S
The EMM metric defined in 2.1 is used to quantify the conditions which empirically trained diffusion models exhibit memorization behavior similar to the theoretical optimum. The selection of Ç determines the closeness between the empirical model and theoretical one. Generally, a smaller Ç requires a higher memorization ratio for the empirical diffusion model to approximate the thcorctical optimum. In Figure 22 we select various Ç in different configurations. We observe that a smaller Ç lypically results smaller EMM. Meanwhile, the conclusions in our main paper remain consistent. For instance, smaller data resolutions, larger model widths, larger numbers of classes for conditioning load to more memorization.
32


Reviews:
[{"review_id": "O1a7p23nwb", "reviewer": "Reviewer_xsGH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. **Significant Advancement in Diffusion Models**: Comprehensive exploration of diffusion models, addressing their capability for high-quality generative tasks across diverse domains such as image, audio, and graph generation.\n2. **Empirical and Theoretical Analysis**: Strong empirical results combined with a robust theoretical foundation, including the derivation of a closed-form solution for the optimal score model.\n3. **Comprehensive Factor Analysis**: Detailed investigation into data distribution, model configuration, and training procedures, and their influence on memorization behaviors.\n\n### Weaknesses\n1. **Focus on Small Datasets**: The study places significant emphasis on smaller datasets, which may constrain the generalizability of its findings when applied to larger, production-scale datasets commonly used in diffusion models. Specifically, the experimental setup is simplified by employing a subset of the CIFAR-10 dataset, limiting its applicability to more complex and diverse real-world scenarios.\n2. **Focus only on single metric**:  The results lack a discussion of other relevant metrics, such as the quality of generated outputs or the convergence of the loss function.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies memorization in diffusion models, introducing the Effective Model Memorization (EMM) metric to quantify conditions that lead to memorization. Experiments show that smaller datasets and random label conditioning significantly increase memorization rates, highlighting privacy risks and offering insights for improving generative model design.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Refine Metric Justifications**: Provide a detailed rationale for chosen thresholds, such as the 90% memorization ratio, and explore the impact of varying these values.\n\n2. **Discussion on real-world scenarious**: the experiments are conducted on very small datasets, raising questions about how these findings translate to real-world scenarios involving large-scale datasets and models trained on millions of samples. Adding a discussion on the scalability and applicability of the results to such scenarios would provide valuable context and strengthen the study's relevance to practical applications.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sOb4BO7244", "reviewer": "Reviewer_td2T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** The paper contains extensive ablation studies and their analysis includes simpler unconditional diffusion-models as well as the larger text-conditional diffusion models. \n\n**Weaknesses:** \n1. In my opinion, the major weakness is that most of these results are expected. For e.g., it is indeed natural and even expected to observe memorization with reduced training datasets. From this perspective, it may seem like the contributions of the paper are very limited. However, the extensive ablation study and experimental findings may be of some interest to future practitioners in this area. \n2. The paper appears to suggest that memorization in diffusion models is solely due to the fact that a closed-form solution of optimal score-function exists. I disagree with this characterization. For e.g., consider normalizing-flow: while the theoretically-optimal flow transformation for a small set of images is not immediately clear, it would almost surely overfit given sufficient capacity. In fact, many findings in this work are not specific to diffusion-model training and are a direct consequence of the supervised-training. \n\nOverall, I do not strongly feel against this paper since the experimental analysis and results are still interesting to know (even if many of these are expected). I am interested to see what other reviewers/AE feel about the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is on memorization in diffusion models and conducts an ablation study as a function of training-data, model-sizes, optimization-configurations and conditioning. To that end, the paper introduces a metric called \"effective-model memorization\" that is defined as the number of training samples for which the model does not generalize beyond the empirical score-estimate (i.e., a non-parametric score-estimator derived using samples in the training-dataset).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not have strong recommendations for revisions. But, I would appreciate it if the authors could address the weaknesses --- for e.g., by pointing out findings that exclusively hold for diffusion-models but not other standard classification/generative models.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eyZZzldRz7", "reviewer": "Reviewer_bw9d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Novel Metric for Memorization: The introduction of EMM provides a concrete way to quantify and explore memorization across different setups.\n* Comprehensive Experiments: The paper evaluates multiple factors, including data diversity, resolution, model size, and conditional labels, providing robust empirical support for its claims.\n* Practical Implications: Insights into privacy risks and training considerations in generative models are relevant for real-world applications.\n\nWeaknesses:\n\n*  Convexity Argument Ambiguity: The claim that Eq (5) is a convex optimization problem requires elaboration, especially since $s_\\theta$​ seems inherently non-convex in complex architectures. This theoretical foundation needs strengthening.\n* Assumptions: The assumptions underlying the theoretical results are not clearly stated, making it difficult to reconcile the theory with empirical observations.\n*  Limited Dataset Variety: Over-reliance on CIFAR-10 may restrict generalizability due to its low resolution and limited size, especially when larger datasets are essential for practical use cases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper examines the phenomenon of memorization in diffusion models. By defining Effective Model Memorization (EMM), the authors systematically study the factors that influence memorization, including data distribution, model configuration, and training procedures. Key contributions include:\n* Establishing the theoretical optimum of diffusion models and highlighting its tendency to memorize training data.\n* Empirical insights into how dataset size, model architecture, and training conditions affect memorization, with CIFAR-10 as a primary dataset.\n* Surprising observations, such as the role of uninformative random labels in triggering memorization.\n* Analysis of memorization across various models and setups.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarify Theoretical Foundations: Elaborate on why Eq. (5) is a convex optimization problem. Please, provide specific conditions or assumptions that justify this claims.\n\n2. State Assumptions Explicitly: Clearly outline the assumptions underlying the theoretical results.\n\n3. Explain which assumptions lead to deviations between empirical and theoretical results. Specifically, what implicit or explicit assumption removes the effect of sample size (and other factors such as steps, training time etc.) on memorization. Why does empirical results diverge from theory? Is the cause primarily due to model capacity, dataset characteristics or some other factor? A brief discussion would help readers.\n\n4. Extend Beyond CIFAR-10 (optional): Include experiments on larger and more diverse datasets (e.g., ImageNet) to validate the generalizability of the findings.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the main contribution of this paper is to provide a nice principled memorization method for diffusion models. The claims of the paper seem to be well supported and the reviewers are supportive of acceptance.
Decision: Accept (poster)