Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis
Authors: Qi CHEN
Jierui Zhu
Florian Shkurti
Venue: ICLR 2025 Poster (2025)
Keywords: Generalization bounds; information theory; generative models; VAE; diffusion models;
Primary area: generative models

Abstract: Despite the empirical success of Diffusion Models (DMs) and Variational Autoencoders (VAEs), their generalization performance remains theoretically underexplored, especially lacking a full consideration of the shared encoder-generator structure. Leveraging recent information-theoretic tools, we propose a unified theoretical framework that provides guarantees for the generalization of both the encoder and generator by treating them as randomized mappings. This framework further enables (1) a refined analysis for VAEs, accounting for the generator's generalization, which was previously overlooked; (2) illustrating an explicit trade-off in generalization terms for DMs that depends on the diffusion time $T$; and (3) providing computable bounds for DMs based solely on the training data, allowing the selection of the optimal $T$ and the integration of such bounds into the optimization process to improve model performance. Empirical results on both synthetic and real datasets illustrate the validity of the proposed theory.

Published as a conference paper at ICLR 2025
GENERALIZATION IN VAE AND DIFFUSION MODELS:
A UNIFIED INFORMATION-THEORETII ANALYSIS
Qi Chen 1,3.4[ Jierui Zhu & Florian Shkurti!:4 14.55
1 Department of Computer Science University of Toronto
2 DDpaatmeet of Statistical Sciences, University of Toronto
3 Data Science Institute, Veecto Institute, 5 Robotics Institute
ABSTRACT
Despite the empirical success of Diffusion Models (DMs) and Variational Au- toencoders (VAEs), their generalization performance remains theoretically under- explored, especially lacking a full consideration of the shared encoder-generator structure. Leveraging recent -eformation-thcorrticc tools, we propose a unified theoretical framework that provides guarantees for the generalization of both the encoder and generator by treating them as randomized mappings. This framework further enables (1) refined analysis for VAEs, accounting for the generator's 's gen- eralization, which was previously overlooked; (2) illustrating an explicit trade-off in generalization terms for DMs that depends on the diffusion time I'; and (3) pro- viding computable bounds for DMs based solely on the training data, allowing the selection of the optimal T and the integration of such bounds into the optimization process to improve model performance. Empirical results on both synthetic and real datasets illustrate the validity of the proposed theory.
INTRODUCTION
Modeling complex data distributions with generative models has become key focus in machine learning, driving transformative applications in various domains such as computer vision (Rombach
al. 2022), natural language processing (Brown et al.) 2020). and scientific discovery (Hoogeboom
et al. 2022). Over the past decade, this field has seen rapid advancement with the emerging of
frameworks such as Variational Auto-Encoders (VAEs) (Kingma & Welling 2013) Makhzani et
2015, Tolstikhin et al., (017). Generative Adversarial Networks (GANs) (Goodfellow al. 2014
Nowozin et al. 2016 Arjovsky et al.) 2017). Diffusion Models (DMs) (Song & Ermon 2019 Song
al. 2020b 2021). well energy- based and auto-regressive models (Larochelle & Murray 2011
Van den Oord et al. 2016). Notably, deep latent diffusion models that combine VAËS and DMs have
recently demonstrated exceptional success in generating righrresolution images (Rombach al.
2022] and videos (Peebles & Xie (203)).
The empirical success and appeal of these models lie in their ability to generate new data rather than merely memorizing and reproducing training samples. However, recent studies have demonstrated memorization problems in GANs (Feng al.
et al., 2020), VAEs (Van den Burg & Williams 2021). DMs (Carlini et al., 2023), ) and auto-regressive Large Language Models (LLMs) (Carlini et al.) 2022), raising significant privacy and copyright concerns. Consequently, understanding the generalization ability of generative models in producing diverse, novel samples becomes critical and urgent. While various generalization theories have been developed for GANs (Arora et al.) 2017, Biau et al. 2021 Mbacke et al. 2023 Yang & Weinan 2022 Ji et al. (021) by adapting different theoretical tools originated in supervised learning (e.g. VC-dimension (Vapnik et al.
1994). Rademacher (Bartlett & Mendelson, 2002), PAC- PAC-Bayes Shawee Taylor & Williamson 1997 McAllester 1998)), their counterparts for VAES and DMs remain comparatively underexplored.
In this paper, we aim to provide novel generalization theory for VAEs and DMs. In comparison to previous works, our main contributions are as follows:
"Correspondence cichen@cs.trronto.. florian tor.otto.co..

---

Published as a conference paper at ICLR 2025
1. Unified information-theoree framework. VAEs employ a probabilistic
pair, while DMs can be viewed as an infinite-length composition of such encoder-generaee sequences (Tzen & Raginsky 2019; Huang al., 202TI. We thereby model the encoder and generator as randomized mappings and develop a novel and unified theory with information-theoreti learning tools, avoiding traditional methods designed for deterministic mappings, which unsuitable for our analysis. The general theoretical results are presented in Sec. 4 where the proposed bounds are algorithm- and data-dependent under the sub-Gaussian assumption.
2. Improved analysis and tighter bounds for VAEs. To the best of our knowledge, we are the first to consider the generalization properties of both the encoder and generator in VAEs. whereas Chénief-Addelatti et al. (202)) only consider guarantees for reconstruction loss and Mbacke et al.) (2024) prove bounds for a fixed generator, ignoring its generalization. Moreover, compared to MMbacee et al. (024), we provide a tighter generalization bound for the encoder by directly bounding the generation error (defined in Sec 3.2), removing the unnecessary Wasserstein-2 2 distance.
3. Computable bounds for diffusion models. We provide non-vacuous upper bounds for DMs to measure the divergence between the generated data and the original data, which can be tractably estimated, as detailed in Theorems 6.2 and 6.3 Through these bounds, we show that:
(a) There is an explicit trade-off between the generalization terms of both the encoder and generator that depends on the diffusion time T'. As presented in Theorem 6.2 when T approaches infinity, the encoder's s generalization term vanishes, while the generator's 's term remains non-zero. Conversely, for small T, the encoder's generalization term dominates. Empirical validation on both synthetic and real datasets verifies this phenomenon. Notably, this result implies that longer diffusion time does not necessarily lead better generalization. To the best of our knowledge, this is the first explicit theoretical formulation of this trade-off in the context of generalization theory.
(b) The proposed bound provides practical guidance for hyperparameter selection, where previous methods fall short due to the difficulty of accurately estimating the divergences between generated and test data, shown in Fig. 3 Additionally, the bound can be estimated using only training data, providing a practical and sample-efficient way to select the optimal diffusion time I or integrate the bounds into optimization for better model performance.
2 RELATED WORK
In this section, we only discuss the most relevant related works, while other related works on diffusion models and convergence theory are presented in Appendix [
Theories for VAEs. Some previous works (Bozkurt et al. 2019: Huang et al. 2020 Bae et al. 2022) analyze VAEs with rate-distortion theory. Another approach involves deriving exact formulae under specific data distributions and high-dimensional limits. Assuming sample size
oo, Refinetti & Goldt (2022) examines the test error for nonlinear two-layer autoencoders as d + oo. Focusing linear 3- VAEs. [Ichikawa VAEE,Ichhikaa & Hukushima
(2024) analyzes generalization error with SGD dynamics, where -xiett analysis reveals posterior collapse when exceeds some threshold, suggesting appropriate KI. annealing to accelerate convergence. [Ichikaw & Hukushima (2023) uses the Replica method to derive asymptotic generalization error for a = d' ((1) showing a peak in eтror at small B. This disappears after [3 exceeds some threshold, leading posterior collapse regardless of the sample size. Husain et al. (2019) build a connection between GAN and WAE, where the generalization analyzed based on the concentration result of Weed & Bach (2019). Abériet-Abbellatif et al. (2022) applied PAC-Bayes theory to derive the generalization bound for the reconstruction loss. Recently, Mbacke et al. (2024) proved statistical guarantees with PAC-Bayes theory. However, their bounds only consider the generalization properties of the encoder. Instead, we provide tighter bounds for the encoder and use cntemmation-theoreen tools to derive generalization bounds for both the encoder and the generator, which are valid for non-linear deep-lcarning models.
Generalization theory for diffusion models. De Bortoli (022) prove statistical guarantees by simply bounding the Wasserstein-1 distance between the population and empirical data distribution without considering the algorithm or training dynamics. [idstrrgach (2022) discuss the errors in the initial condition and drift terms for SDEs used in the DMs, illustrating the drift explosion under the manifold assumption. They also propose that to avoid purely memorizing data, the exponential integral of the drift approximation emror introduced by the score function must be kept infinite when
2

---

Published as a conference paper at ICLR 2025
minimizing the score- more-matching loss. However, their results are not quantitative. The most recent work al. (224)) studies the generalization properties of DMs with the random feature model, extending results in Song et al. (2021) by providing separate generalization analysis for the gott-ng loss. Its bound cannot capture the trade-off on diffusion time, which was introduced via an BLBO decomposition of the training loss of diffusion models by |Franzese et al. (2023). However, our work is the first to show that it exists in generalization, as well as why. Our theoretical results differ from all the mentioned approaches above. We leverage informatiiontthorrtti tools to obtain algorithm- and data- data-dependent bounds under nab-Gaussian assumption, in contrast with the data data-dependent bound in Li et al. (2024) that assumes the target data distribution is a 2-mode Gaussian mixture, The proposed bound can provide non-vacuous estimation for the divergence between the original and the generated data distribution, demonstrating an explicit trade-off on the diffusion time.
3 PROBLEM SETUP
Notations. Detailed notations are summarized in Table T We use upper case letters to denote random variables (e.g., X, Z) and corresponding calligraphic letters X' Z to denote their support sets. We write (X)) as the set of all the probability measures over X. Then, we denote Px E (X)) as the marginal probability distribution of X. Following Husain et al. (2019). we further use
2) def {f: + 2} to denote the set of all the measurable functions from X' to Z. For any F(X, 2), the pushforward distribution of Px through f is denoted P// def f# Given Markov chain X + Z, we use P22X to represent the distribution over a space Z conditioned on elements from X. which is also known as the Markov transition kernel from X' Z. We adopt similar notations for the Markov chain in reverse direction Z + X but using Qz, Qx z to make a distinction. Let (-Il-) denote the divergence between two distributions. To be used in our paper, we recall the definitions of Wasserstein distance, the Kullback-Leibler (KL), Pensen-Shannon (JS), and Fisher divergences in Appendix A.2 For any positive integer m, we denote [m]
(2).
def {L.....m).
3.1 GENERALIZED FORMULATION
Deep generative models typically transform simple, casy-to-sample prior distribution over latent space 2 into a target data distribution defined on the input space X' though a generator (X)) In most cases, the input and latent spaces subsets of Euclidean spaces, i.e., X Rd and 2 C Rdz Ideally, when applied to an casy-to-sample prior distribution Qz (e.g., a Gaussian), the optimal generator will induce identical distribution as the target data distribution Px. However, analogous to the no free lunch theorem in supervised learning (Shaley-Shwartz & Ben- too complex to be learnable without any prior knowledge, so we further suppose the learning is conducted within a generator hypothesis
G: 2
David 2014). the set of all possible generators F(Z, P (X))
def G#Qz to Px, by minimizing their divergence (( xx(()) For example, GAN considers the aforementioned goal by directly solving inf D [Px|
set 9 C F(2, (X)). The primary goal is to learn a G that matches Q
Geg
However, directly learning G may be hard and instable for some data distributions. In this paper, we focus on VAEs and diffusion models, which indirectly learn G the inverse process of an additional probabilistic encoder E, sharing similar mncoder-generall paradigm.
Eenoeeer Structure. Let us define the encoder hypothesis set & C F(X, P()). Then, the encoder is a function E : X' + P (3), E E E that maps an input data point X ~ Px to conditional distribution over the latent space Z, i.e., E(X) can be alternately denoted as P at
the population level. We further denote the probability densities of E(X) and PE E# Px as (z(2(z) and (z))), respectively. Similarly, the respective probabilistic densities of G(Z) = Q and G#Qz denoted ()(=)2) and qG(x), The above definitions cover the original auto- encoder, where E' is a deterministic encoder by restricting È as the set of delta distributions. Since DRI(Px (G))2z < en [ID KL (Px x [l x G(Z))] with data processing inequality, the objective can be relaxed to solve the upper bound:
inf DKI(Px x E(X)IlQ x G(Z)). Geg
(1)
3

---

Published as a conference paper at ICLR 2025
Furthermore, show in Appendix B that VAEs and score-based DMs, respectively, optimize this objective's ' two forms of decomposition.
3.1.1 VARIATIONAL AUTO-ENCODER (VAE)
Based on the general objective, VAE uses decomposition that is equivalent to the common variational inference approach (Kingma et al. 2019), where the optimization objective is proportional to:
inf GEG.EEE
[x (Bz~E(x) [- [x]])() + (Ex [xx(
Constraining E and G' to specific distribution families leads to the traditional VAE objective. In VAEs, the latent space typically has a much lower dimensionality than the input space, with d2 < d1-
3.1.2 DIFFUSION MODEL. (DM)
As discussed in (Huang et al. 2021 Izzn & Raginsky 2019; Kingma et al., 2021). ; one can treat DMs infinitely deep hierarchical VAES by sequentially composing N probabilistic encoders
E def [= and generators G1.N {G+ where N + oc. In another view, we can consider the encoders and generators as time-dependent randomized mappings that directly applied to the original data distribution and the latent prior distribution, respectively. The difference is shown Fig. I We further provide a detailed comparison of these two viewpoints in Appendix B.2 for hierarchical VAEs and in Appendix B.3 for the other.
Hierachical VAE
Px1
Pxs
PIw
~
Q
Qix-
Qxi
Q
E2
EN
G1
GN-1
ON
Time lime-dopendent Mapping
Gv
Ex
Gx-1
Eg
E1
G1
Px:
Px
Xx
Pxx
Q
xt
Q
ix--
Q
Q %b
Figure 1: Illustration DMs as hicrarchical VAEs and time-dependent randomized mapping
Discrete-time stochastic process. Without loss of generality, we consider the latent space the same the input space with d1 d2 d for typical DMs, where E& X + P (X),, G3 X 9 (X). For a forward discrete-time stochastic process, we denote the marginal distribution at step k as Px.- E& (Xo), X, Px, and Px. = # Px,, where Px, Px is the initial data distribution. The forward process is often set to achieve some easy-to-sample noise
Then, we have Vk [N], [N].X, Xxx
distribution ı (e.g., π AO Ia)) where Pxxx π, and Pxx N-000, almost surely. Conversely, the backward process starts from Qxy and aims to achieve Q z Px Then, we denote the marginal distribution at step k for the backward process introduced by the the generator sequence G1.N Q Qx Gx#Q&
Continuous-time diffusion process. As Song al. (020b), in the continuous-ting limit, the above forward process can form diffusion process X, fi=u solving the following SDE over diffusion time
dx, s(X,, ()dt X())W Xo Px,
where we have Px, Pxo f(.,t:: X is the drift coefficient, X(t) is the diffusion coefficient, and {We)eejorr) is a Wiener process. With the appropriate selections of and X, the above SDE can converge the predetined prior distribution ा.
Haussmann & Pardoux (1986), there exists reverse-time diffusion process {Xe He satisfying (X,77 (X,)7- under mild conditions, which is the solution from t = T to 0 of the following SDE:
According to Anderson 1982):
dx, = (x(,.))
A())W, Ny Pxr.
4

---

Published as a conference paper at ICLR 2025
The above ideal backward process can be used to generate reference sample sequences to learn the generative dynamics, and we denote the ideal generator as E vt E [0, TV. Then, by approximating V log Pe (X,) with V log q: (X,,, the generator G.. Vt E [0, 7) is characterized by the following SDE:
dx, - (f(,, A(t) log qu (x, )t + A(t)dW NT
(5)
def Define 0x--1 GT#T as the generated distribution at time Song al. (2020b 2021 further
proved that, under some regularity conditions, the KL-divergence between the real data distribution Px and the final-time generated distribution Gr## is bounded by:
1 ["' (2 (Px, 11Q )dt Dxx (KL(Px S 3==0
where the Fisher divergence is defined in Def. A.2 As we show in Appendix B this is an upper bound of another possible decomposition of the general objective Eq.
3.2 SETUP FOR GENERALIZATION ANALYSIS
So far, have discussed the learning objectives of VAEs and DMs at the population level. However, due to the unknown nature of Px. these learning objectives can only be estimated with a training dataset S {X+) of m examples, where each X; Px and S Px The empirical distribution of these observations is then denoted Px def 1 E 5x,- By optimizing the encoder E and generator G w.r.t empirical learning objectives, they are learned from the data and mutually dependent. Intuitively, One can as consider them as respectively approximating the posteriors Pz x,5
m
and Q x|
The encoder- encoder-generator process may overfit the empirical distribution Px. particularly when the number of training examples is limited. To measure the performance gap between the population and empirical objectives for generative models, we further define the loss for the encoder-generator
pair as Âg X' x Z x X' + R In particular, we use Aa X) || - X in Corollary. 4.2
and A X, X) = log qa (X (2) in Corollary. 4.3
Let us first consider the empirical reconstruction error ! by which we can measure the input-output distortion in expectation to the empirical measure Px of the train dataset
px (E, G)
E Ez Ez-E(X,) Ex- (, X,X.).
Now, we define the generation error the expected difference between any input X sampled from the data distribution Px and any output generated by G(Z)) with being sampled from the prior w:
c7 (E, G) def
Ao Z.X)
The dependence on encoder E' is implicit and specific to the encoder-generate paradigm, where the learning of G relies on E. Based on the definitions above, introduce the generalization gap that measures the difference between the generation error and the empirical reconstruction error:
Ac gen Px
(E, G, del
Es-Pp (E,G) - C (
Intuitively, when the encoder and generator are learned from an empirical reconstruction process with very small reconstruction error (i.e., L ir (E, G) is small), the generalization gap reflects the average difference between the generated and original data.
4 GENERAL THEORETICAL RESULTS
In this section, we present general theoretical results for generative models that share the same encoder-generatee paradigm, which can be directly extended to analyze models like VAEs and DMs.
Similar notion distortion be found
Blau Michaeli 2019
5

---

Published as a conference paper at ICLR 2025
Theorem 4.1. For any encoder E E and generator G E 9 learned from the training data S = {Xi] assume that the loss ( X) is
(See definition in Def. A.4
under Px,2xx = Qxi2 x Oz X Px, where Z ~ Qz -T G(Z), X, Z are respective
independent copy of X and Z. such that X, % 11. X. Then, VX, E E(X,), X; G(Z,), the generalization gap admits the following bound:
V2R m
Ao ge (E, GT
Σ
[x,
I(x; 2.)
Discussion. Since Va +b S va + Vo, Va.b O. the above bound can be further decomposed
as V/R m Em= Ex, [DL (EX)(( + V2R E. / (X, (x The first term measures, on expectation of randomly drawing m data points, the average divergence from their encoded latent distributions to the predefined prior æ, which reflects the generalization of the encoder. The condition mutual information in the second term [X,; 12, measures the generalization of the generator G.
This bound provides insight into how the generalization of the encoder and generator interact. Assuming the reconstruction error is small: If the first term approaches zero, i.e., E(X,) n, which means Z; contains no information of the training data, the generalization gap is entirely attributed the generator. In contrast, if the encoder overfits the training data and Z; fully captures the information from X,, then [X,;; X,| = making the second term zero. In this case, the generalization gap is entirely due to the encoder. In intermediate scenarios, both the encoder and generator contribute to the overall generalization. The detailed proof can be found in Appendix C.
By specifying ÂG. we have the following two corollaries that measure the divergence between the true data distribution Px and the generated distribution G#T, formulated in Sec. 3.1
Corollary 4.2. Under Theorem 4.1 let Ao(X, X) = 1| X].. Then, the Wasserstein distance between the data distribution Px and the generated distribution G# is upper bounded by:
V2R
VVX
Dw, (Px(IG#T) S BsL. (E, G)
[ T(X;; (i1ZA)
m
Corollary 4.3. Under Theorem 4.1. let the density function of probabilistic decoder G given a latent = x)=
code a be qG(-12) X + R2J and Ao(X, 7, X)
log 9a (X) ()). The KI.-divergence between the data distribution Px and the generated distribution G# π is then upper bounded by:
V2R
DRL ORI(PXI)G < E2C (E,
[xx + X114-
T7L
where M(Px)= = Ex| log (e)p(()) X denotes the entropy.
The proofs of these two corollaries are presented in Appendix C.2 In following sections, will
apply Corollary 4.2 and Corollary 4.3 to establish Theorem 5.1 and Theorem 6.2 respectively.
ANALYSIS OF VAES
VAEs specify tractable distributions both encoder and generator. Normally, they set as Gaussians, as presented in the original VAE (Kingma & Welling 2013). Using some neural networks parametrized by Φ. one can map the original data the latent space and model both the mean and variance of the Gaussian as Ma : X' + Z and 0 : X + Z. The encoder is then Es(x) = Analogously, the generator network is parameterized by 0, which often only models the mean. where Gø(z) = (z)(2), Ia.) with pe : Z X. Directly applying Corollary 4.2 we can obtain the following bound for VAE:
(H)(()), diag(o)(z)))
Theorem 5.1. Under the assumptions made in Theorem 4.1
4.Iaa and Corollary 4.2 we have VX, E S, Z4 ~ Ex(X,), X₂ ~ Go (Z,) over the draw of m samples with S ~ Px that the following bound holds for any probabilistic encoder Eig and generator Go defined above:
V2R
m
Dw, (Px Go S EgL (Es, Go) +
Ex, Ex.Dx((F((X))I)) (xX
m

---

Published as a conference paper at ICLR 2025
Comparison with previous VAE bounds. The above bound could be compared to the recent PAC- Bayes bound for VAE either by converting to a high-probability bound using the results of Theorem 3 Xu & Raginsky (017)) by converting PAC Bayes bound to its expectation version. In sharp contrast to the bound Theorem 5.2 of Mbacke et. (024), which applies only to a fixed 0, the above result guarantees generalization for any generator Go with Ei [(x;; Zz). Additionally, our approach avoids introducing an extra Wasserstein-2 distance,
directly bound the generation error using the empirical reconstruction loss. In contrast, Mbacke et al. (2024) utilize the triangle inequality
for the Wasserstein distance, separately bounding Dw; (Px I 22 and Dw, (Go# Pz ||G #r) Furthermore, we relax the strict assumption of bounded support, replacing it with the more flexible uub-Gaussianity condition. Detailed mathematical and experimental comparisons are respectively provided in Sec. D.I [ and Sec in the Appendix, showing the improvements.
Insights and practical guidance for VAEs. If we instead apply Corollary 4.3 the reconstruction error becomes fxx (Fig, Go)
Ei (x,,-Exx.)) log qG, (Z)] Interestingly, jointly opti- mizing it with m Σ1=1 D DRI. (E(,) )T) yields the empirical estimate of the VAE objective in Eq. (2) This implies that the VAE training process inherently accounts for the encoder's 's generalization. How- ever, the generalization of the generator G is often overlooked. Therefore, potential improvement could involve explicitly incorporating the generator's generalization into the optimization objective as a regularization term. In Appendix [.22 we derive an example regularizer using upper bound of the conditional mutual information term by introducing additional randomly initialized generator.
6 ANALYSIS OF DIFFUSION MODELS
In existing literature, most works focus on the convergence of DMs, while the limited analysis of their generalization properties typically involves separately bounding D Pxxx and DPxx|GG #*) using concentration results, then combining them via the triangle inequality. However, widely used divergences in diffusion models, such as KL-divergence, do not satisfy the triangle incquality. Moreover, these bounds are often loose due strong assumptions about the data distribution, score function estimation, and insufficient consideration of the learning algorithm. To address these, we directly bound ID (P'x #T) and apply information-theoreii tools derive computable algorithm- and data-dependent bounds in this section.
6.1 GENERATION ERROR BOUND FOR SCORE- BAREE DMs
For DMs, the encoders and generators E,, Gr A7 E [0, 7] are restricted the family of SDEs the discretized Langevin Dynamics. Before bounding the generation error, we first prove the following lemma (proof in Appendix E.T) on the empirical reconstruction error at the diffusion end time I:: Lemma 6.1. Let {X,TT be the empirical version of the forward diffusion process defined in Eq. 3 where Xo Px. We assume the existence of the backward process under the regularity conditions outlined in Song et al. (2021) and denote it as {X:){= (X.)= which results from the reverse-time SDE defined in Eq. (4. Then, the generative backward process (X,) (T))T_ o is defined
in Eq. (5). Let E,, E, ,Gt, Vt [0, T be their corresponding time- dene-dependent Markov kernels. The
density function of any generator G. given latent code 2, is denoted qa(-12) X - R. and let Ao(X, 2, X) -log dogga(( (X|Z)) Then, we have
1 1 (P(I)Drishen (px 11 )dl. Pc (Er. Gr) - Lp (Er, ET S ==
Connection to score matching, By setting the derivative of the log density of Qx, with some parameterized function, i.e., Vz log (z) = s0(x, t), the upper bound in the above theorem gives the following empirical loss of Explicit Score Matching (ESM):
[ESM(O, 1 /" Ex,~xx (Ok=P,,D((111))) P/ x, logı:(X.) - (x( ( li
Since (X,) m
Em PE, (x), it gives the following Denoising Score Matching (DSM) loss:
ΣDSM (0, ((.)
E Px (X.) (1)]l x, log,E, Pe. (X,) so (Xx, t) 1)]
==
7

---

Published as a conference paper at ICLR 2025
which is equivalent ESM up some constant discussed in (Vincent 2011
Combining Corollary
4.3 and Lemma 6.I we have the following generation error bound for score- score-based diffusion models:
Theorem 6.2. Under Lemma 1.1 for any SDE encoder E, and generator G. trained via score match- ing on S = (X; j== the corresponding outputs at the diffusion time T are XT (X,). Xo (XT) for each X; E The KL-divergence between the original data distribution Px and the generated data distribution G4 # diffusion time T is then upper bounded by:
(PL(Px|G77 & Es
(X.) (aax)&CEssee LESM (0, 2(1)
V2R m
/R
m
E xx DKL
m Σ
X(
m
22
Ts
The proof is presented in Appendix E.2 Compared to recent work on the generalization of DMs al. 2024), we demonstrate the existence of a trade-off w.r.t the diffusion time T. In contrast to Franzese et al. (2023) that also mentioned the diffusion time trade-off, we prove explicit form of the tradeoff related to generalization terms, whereas Franzese et al. (2023) only justified via a new ELBO decomposition of the training loss at the population level, without being able to show how it affects generalization.
Explicit trade-off on diffusion time T. The KL-divergence terms in the above bound reflect the generalization of encoder Er. Since Ti < 0 and Tr > O, there exists an inherent trade-off on the diffusion time T when minimizing the two terms. Note that when T - oc, the forward SDE maps the empirical data distribution to the noise, which means #x will converge n. This makes the two KI. terms T, and T2 in the above theorem equivalent, and both will converge to zero, as discussed in See 6.2 However, T3, which characterizes the generalization of generator Gr. will remain non-zero for small sample size m. We bound T3 in Sec. 6.3 to a easy-to-compute form, showing a linear growth w.r.t T. This means another trade-off between the generalization of the encoder and that of the decoder exists on diffusion time.
6.2 GENERALIZATION FOR SDE ENCODERS
The typical encoder of diffusion models is set to a special class affine SDEs that has closed-form solution (Särkkä & Solin 2019), where d.X, (t) X()dW,, Then, the encoder posterior for given example equals to (X,) ((T)
where r(I) e a(t")dt' v())
V I (f))
Variance-exploding SDEs with parameter att) = 0, A() /do)) o²(() = noaaa// m have (X)), = (X,,, (T) - o² Lo)) which do not converge to a steady-state distribution because the variance grows. Setting the prior = (N(0, (o² x² (O)) we have
(xr(Br(X.) = (x/ X o0)) T- O.
Variance-preservingg SDEs converge to multivariate Gaussians with at)) = (t), = /Bo + (81 Bo)t. By denoting $T 3,T + [ - B.)T the encoder posterior equals to (x,),) = -N(e br X,, (1 e (r L.). Setting the prior as Ia), we have
(E
(e Br X² - fr dlog(1 - c )-))) T 0.
6.3 GENERALIZATION FOR DISCRETIZED SDE GENERATORS
We focus on the generalization terms and drop the analysis of the convergence w.r.t the score-matching training process, which has been done in Li et al. (2024). The generalization gap term related to the generator G.. is determined by I (X X, Xr), where 9 is learned from data and can be represented some function of the train dataset.
8

---

Published as a conference paper at ICLR 2025
Theorem 6.3. Let the step size be T N. where split T to N discrete times. For any [N], we use the following discrete update for the backward SDE by setting Ct2 ~ (O,Io),,t, tx T Tk:
Xt. = (1 X (x-1)) + + - tR - -1 - -t=-) VFX(T 1x-1)EtZ-1
Furthermore, we assume a bounded score | log (r(()) S L, Vx, Then, we have
TL Σk=1 X²( 1R-17
m Σ x(xox)x,177>
1 r(Xo: X1mm (T)
m
2mN
The proof is deferred to Appendix F This theorem can be used to estimate Ts. which has a lin- ear growth w.I.t T for variance preserving SDE. Combining Theorem [.2] we obtain the sample complexity (1/)mm), compared to (m 2/6) in Li et al. (2024) using the random feature model.
Practical guidance for DMs. Since the upper bound in Theorem 6.2 can be estimated with only training data, we can train the model for various diffusion times, estimate the bound, and select the optimal T via grid search. Additionally, the generalization terms in the theorem can be incorporated regularization when optimizing the score-matching model parameter 0. This can be achieved by selecting appropriate values for So. 31 in X(t), or by adding gradient penalty to control the Lipschitz constant of the score model sg.
EXPERIMENTS
In this section, we focus validating Theorem 6.2 for score-based DMs using both synthetic and real datasets. The numerical results illustrate the existence of the trade-off between the generalization terms of encoder and generator on diffusion time, which significantly impacts generation performance. Experiments for Theorem 5. VAEs are deferred to Sec G. in the Appendix, where we show the proposed bound can better capture the generalization of VAΣs compared to previous bounds with the additional mutual information term for generator, and its relation to the memorization score proposed in Van den Burg & Williams (02TT.
7.1 SYNTIETIC DATA
We begin by validating the theorem on a simple synthetic 2D dataset derived from the Swiss Roll dataset. We train the score matching model s0 and estimate the upper bound in Theorem 6.2 on a training set of size m. W.r.t the expectation over dataset S, we conduct 5-times Monte-Carlo estimation by randomly generating train datasets with different random seeds. For the left-hand-side KL-divergence, we conduct Monte Carlo estimation of with 1000 test data points. To get more details on the estimation, please see Appendix G.2.
teel datle KL. uppor bound
Bound
260
1000
1000
3.48
435
140
Number Examples::
(b)
Time
Figure 2: Evolution of bound and dast-data KL--ivergence estimation w.r. (a) train dataset size m when diffusion time is T = 1, (b) diffusion time T when the training dataset size is m 200.
Sample complexity. We can observe in Fig. 2 a) that both the estimated test-data KL divergence and the upper bound decrease with the increase in train dataset size m, corresponding to the diminish of Ta with order ((1//m) as m + 00. However, they will not converge to zero, which is due to the diffusion time T = 1 # oo with non-zero Tı, T2 and the optimization error of score matching loss. The quality of generated data for models trained on different sample size m aligns with human perception, as presented in Fig. 5 Appendix.

---

Published as a conference paper at ICLR 2025
Trade-off diffusion time. In Fig. (b). can observe the trade-off diffusion time T both the bound and the estimated KL divergence. This indicates the proposed bounds are non-vacuous and capture the algorithm and data distribution well. In addition, the optimal diffusion time lies in the range of 0.4 to 0.6. We visualize the generated data for each diffusion time in Fig. i the Appendix, and observe the generated data points that fit the test data best in human perception also take values at I" = 0.4 or T = 0.6.
7.2 REAL DATA
We further estimate the bound and the test data KL divergence (or log densities) by training DMs on MNIST and CIFARIO datasets with few-shot data (m = 16) and full train dataset. For the full data setting, in Fig. (c), we can observe the trade-off on both the estimated bound using training data and log- likelihhodd (in bit per dimension (BPD)) estimated
10000 test data points. For the few-shot scenario, we observe a trade-of off between noise and duplicate (or entirely black/white) images in the generated data with respect the diffusion time 7 across both datasets, shown in Fig. 3 (a). In 3 (b), the estimated bound can verify the diffusion time trade-off, where the optimal is around 0.8 for both datasets, more detailed results are presented in Appendix (Fig. and Fig. TOP. However, the test data KL divergence and fog-likelihood do not reflect the trade-off. This is consistent with the conclusion in Theis al. (2015) that it's challenging obtain accurate estimation of the KL. divergence and the BPD for high-diman -igensimensional data distribution with limited data.
L Da
(a)
BPD & Test Data KL
spd (mradd
Estimatedd Bound
lpd al) bound (mnag)
Extimated Bound
data mmniet bound (mmid))
150
BPD
10
-50
20.0
BPD & Test Data KL
Estimated Led Bound
17.5
bad (cdtr12) upper bound (cPar)
Estiraleed Bound
-203
15.0
12.5
lpd (d(ar10)
305
tedt data Kl Initert
upper la (other
22
2.00
1044
(b)
Diffusion Time: T
(c)
Diffusion Time: T (log scale)
Figure 3: (a) For DM trained on few-shot MNIST and CIFARIO data (m 16): The trade-off on diffusion time reflects on the generated images with the growing of T (b) The bounds estimated on train data, KL divergences, and log densities (bpd-bit bits per dimension) estimated (c) The bound estimated on train data and log densities estimated on 10000 test samples for DM trained full MNIST (m 60000) and CIFAR10 10 dataset (m 50000).
100 test samples.
CONCLUSION
In this work, we provided unified information-theoreii analysis for encoder-gennerator-yye genera- tive models, offering a better understanding of their generalization properties. Our results improved the analysis of VAEs, provided meaningful generalization bounds for DMs, and explicitly unveiled the trade-off on the choice of the diffusion time T. Empirical validation on both synthetic and real data verifies theoretical results. For discussion of limitations and broader impacts, Appendix H
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We appreciate constructive feedback from anonymous reviewers and meta-reviewers. Qi Chen is supported by grant number DSI-PDFY3RIPI1 from the Data Sciences Institute at the University of Toronto. Florian Shkurti is supported by the Canada First Research Excellence Fund (CFREF).
REFERENCES
Pierre Alquier et al. User-friendly introduction to pac-bayes bounds. Foundations and Trends@ in Machine Learning, 17(2):174 303, 2024.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):3133326, 1982.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Interaational conference on machine learning, pp. 214-223. PMLR, 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International conference machine learning, pp. 224-232. PMLR, 2017.
Juhan Bae, Michael R Zhang, Michael Ruan, Eric Wang, So Hasegawa, Jimmy Ba, and Roger Grosse. Multi-rate vae: Train once, get the full rate-distortion curve. arXiv preprint 1rXi::221..03905, 2022.
Arpit Bansal, Eitan Borgnia, Mong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. Advances in Neural Information Processing Systems, 36, 2024.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482 2002.
Gerard Biau, Maxime Sangnier, and Ugo Tanielian. Some theoretical insights into wasserstein gans.
Journal of Machine Learning Research, 22(119)::-45, 2021.
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The aite-distorrtionpereeptii
tradeoff. In International Conference on Machine Learning, pp. 675-685. PMLR, 2019.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
Alican Bozkurt, Babak Esmaeili, lean-Baptiste Tristan, Dana Brooks, Jennifer G Dy, and Jan-Willem van de Meent. Rate-regularization and generalization in vaes. arXiv preprint
arXiv:1911.04594, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yuheng Bu, Shaofeng Zou, and Venugopal V Veeravalli. Tightening mutual information-basee bounds on generalization error. IEEE Journal on Selected Areas Information Theory, 1(1): 121-130, 2020.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07666, 2022.
Nicolas Carlini, Jamie Hayes. Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 2250,5270, 2023.

---

Published as a conference paper at ICLR 2025
Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pp. 4735 7733-4763. PMLR, 2023a.
Qi Chen and Mario Marchand. Aggorthmm-depender bounds for representation learning of multi- source domain adaptation. In International Conference on Artificial Intelligence and Statistics, pp. 10368-10394, PMLR, 2023.
Qi Chen, Changjian Shui, and Mario Marchand. Generalization bounds for meta-learning: An information-theoretic analysis. Advances in Neural Information Processing Systems, 34:25878- 25890, 2021.
Qi Chen, Changjian Shui, Ligong Han, and Mario Marchand. On the stability-plasticity dilemma in continual meta-learning: Theory and algorithm. Advances in Neural Information Processing Systems, 36:27414 444468 2023b.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215 2022.
Badr-Ed Badr-Eddine -ndrief-Abdellattf,, Yuyang Shi, Arnaud Doucet, and Benjamin Guedj. On pac-bayesian reconstruction guarantees for vaes. In International conference artificial intelligence and statistics, pp. 3066-3079. PMLR, 2022.
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprini 1rXiv:2208.05314 2022.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34: 34:17695-17709, 2021.
John Duchi. Lecture notes for statisties 31 /electricaa engineering 377. URL: https://stanford edu/class/stats3 I/Lecturesffull _otes. pdf. Last visited on, 2:23, 2016.
Qianli Feng, Chenqi Guo, Fabian Benitez-Quiroz, and Aleix M Martinez. When do gans replicate? on the choice of dataset size. Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 6701-6710, 2021.
Giulio Franzese, Simone Rossi, Lixuan Yang. Alessandro Finamore, Dario Rossi, Maurizio Filippone, and Pietro Michiardi. How much is enough? a study on diffusion times in score-base generative models. Entropy, 25(4):633, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Mahdi Haghifam, Gintare Karolina Dziugaite, Shay Moran, and Dan Roy. Towards unified information--hherrtii framework for generalization. Advances in Neural Information Processing Systems, 14:26370-26388 2021.
Mahdi Haghifam, Shay Moran, Daniel M Roy, and Gintare Karolina Dziugiate. Understanding generalization via leave-one-out conditional mutual information. In 2022 IEEE International Symposium on Information Theory (ISIT), pp. 2487-2492. IEEE, 2022.
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kumpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He. Yuxiao Chen, et al. Proxedit: Improving funing-free real image editing with proximal guidance. In Proceedings the the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4291-4301, 2024.
UIrich G Haussmann and Etienne Pardoux. Time reversal of diffusions. The Annals of Probability, pp. 1188-1205, 1986.
Fredrik Hellström and Giuseppe Durisi. A new family of generalization bounds using samplewise evaluated emi, Advances in Neural Information Processing Systems, 15:10108-10133 2022.
12

---

Published as a conference paper at ICLR 2025
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 2020.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation 3d. In International conference on machine learning, pp. 8867-8887. PMLR, 2022.
Chin- Wei Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective diffusion- generative models and score matching. Advances in Neural Information Processing Systems, 34: 22863-22876, 2021.
Sicong Huang, Alireza Makhzani, Yanshuai Cao, and Roger Grosse. Evaluating lossy compression rates of deep generative models. In International Conference on Machine Learning, pp. 4444 4454.
PMLR, 2020.
Inbar Sasberrnnn-Sieeglglass Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12469-1244 12478, 2024.
Hisham Husain, Richard Nock, and Robert C Williamson. Adversarial networks and autoencoders: The primal-dual relationship and generalization bounds. arXiv preprint (902,.9902.00000 2019.
Yuma Ichikawa and Koji Hukushima. Dataset size dependence rate-distortion curve and threshold of posterior collapse in linear vac. arXiv preprint arXiv:2.30.000 07666669007,, 2023.
Yuma Ichikawa and Koji Hukushima. Learning dynamics in linear vae: Posterior collapse threshold, superfluous latent space pitfalls, and speedup with kl annealing. In International Conference on PMLR, 2024.
Artificial Intelligence and Statistics. pp.
Kaiyi Ji, Yi Zhou, and Yingbin Liang. Understanding estimation and generalization error generative adversarial networks. IEEE Transactions on Information Theory, 67 2021.
Ioannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus, volume 113. springer, 2014.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Diederik Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114 2013.
Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends@ in Machine Learning, 12(4):307-392, 2019.
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 29-37. JMLR Workshop and Conference Proceedings, 2011.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, pp. 945-985. PMLR, 2023.
Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization error bounds of noisy gradient methods for non-convex learning, arXiv preprint 1eXiv:1902.0062. 2019.
Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian. On the generalization properties of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv: 1.05644, 2015.
13

---

Published as a conference paper at ICLR 2025
Sokhna Diarra Mbacke, Florence Clerc, and Pascal Germain. Pac-bayesian generalization bounds for adversarial generative models. In International Conference on Machine Learning, pp. 24271- 24290. PMLR, 2023.
Sokhna Diarra Mbacke, Florence Clere, and Pascal Germain. Statistical guarantees for variational autoencoders using pac-bayesian theory. Advances in Neural Information Processing Systems, 36, 2024.
David A McAllester. Some pac-bayesian theorems. Proceedings of the eleventh annual conference on Computational learning theory, pp. 230-234, 1998.
Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. A non-parametric test detect data- copying in generative models. In International Conference on Artificial Intelligence and Statistics, 2020.
Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information- thhoorrticc generalization bounds for sgld via data-dependent estimates. In Advances in Neural Information Processing Systems, pp. 11015-11025, 2019.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
Yookoon Park, Chris Kim, and Gunhee Kim. Variational laplace autoencoders. In International conference on machine learning, pp. 5032-5041. PMLR, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4195-4205, 2023.
Ankit Pensia, Varun Jog, and Po-Ling Loh. Generalization error bounds for noisy, iterative algorithms. In 2018 IEEE International Symposium on Information Theory (ISIT), pp. 546-550. IEEE, 2018.
Jakiw Pidstrigach. Score-based generative models detect manifolds. Advances in Neural Information Processing Systems, 35:35852-35865. 2022.
Maria Refinetti and Sebastian Goldt. The dynamics representation learning in shallow, non-linear autoencoders. In International Conference Machine Learning, pp. 18499-18519, PMLR, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Daniel Russo and James Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory, 66(1):302-323, 2019.
Oleh Rybkin, Kostas Daniilidis, and Sergey Levine. Simple and effective vae training with calibrated decoders. In International conference on machine learning. pp. 9179-9189, PMLR, 2021.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithums Cambridge university press, 2014.
John Shawe- Taylor and Robert C Williamson. A pac analysis of a bayesian estimator. In Proceedings of the tenth annual conference on Computational learning theory, pp. pp. 2-9, 1997.
Alexander Shekhovtsov, Dmitrij Schlesinger, and Boris Flach. Vae approximation emoor: Elbo and exponential families. arXiv preprint 2x2.v:2/02.09) 2021.
14

---

Published as a conference paper at ICLR 2025
Wenxian Shi, Hao Zhou, Ning Miao, and Lei Li. Dispersed exponential family mixture vaes for interpretable text generation. International Conference on Machine Learning, pp. 8840-88511 PMLR. 2020.
Changjian Shui, Qi Chen, Jiaqi Li, Boyu Wang, and Christian Gagné. Fair representation learning through implicit path alignment. In International Conference on Machine Learning, pp. 20156- 20175. PMLR, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265, PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415-1422, 2021.
Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Conference Learning Theory, pp. 3437-3452. PMLR, 2020.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. arXiv preprint arXiv: 1 1.0184, 2015.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprini arXiv: [711: 1711001518.. 2017.
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv: 09888, 2019.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-hased generative modeling in latent space. Advances in neural information processing systems, 34:1 1287-11302. 2021.
J. J. Van den Burg and C. K. I. Williams. On memorization in probabilistic deep generative models. In Advances in Neural Information Processing Systems, volume 34, 2021.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Vladimir Vapnik, Esther Levin, and Yann Le Cun. Measuring the ve-dimension of a learning machine. Neural computation, 6(5):851-876, 1994.
Sergio Verdú. Empirical estimation of information measures: literature guide. Entropy, 21(8):720, 2019.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compula- tion, 2367)11661-1674, 2011.
Ziqiao Wang and Yongyi Mao. Tighter information--heorrtic generalization bounds from supersam- ples. arXiv preprint 23024 2023.
Jonathan Weed and Francis Bach. Sharp asymptotic and finite-sample rates of convergence of empirical measures in wasserstein distance. 2019.
Xuetong Wu, Jonathan Manton, Uwe Aickelin, and Jingge Zhu. An information-thcoreti analysis for transfer learning: Error bounds and applications. arXiv preprint 2rXiv:2207.05377, 2022.
15

---

Published as a conference paper at ICLR 2025
Aolin Xu and Maxim Raginsky, coormattion-heorettii analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems, pp. 2524-2533, 2017.
Gezheng Xu, Qi Chen, Charles Ling, Boyu Wang, and Changjian Shui. Intersectional unfairness discovery. arXiv preprint arXiv:2405. 2024.
Hongkang Yang and Weinan. Generalization and memorization: The bias potential model. In Mathematical and Scientific Machine Learning, pp. 1013-1043, PMLR, 2022.
16

---

Published as a conference paper at ICLR 2025
A PRELIMINARIES
A.1 NOTATION TABLE
Table Summary of major notations
Symbol
Meaning
{ m
Upper letter (e.g. Y)
Random variable
Calligraphic letters (e.g. }
Support sets random variables The set of the probability measures over y
PQ)
Py
Marginal distribution of
X
Input space
2
Latent space
2)
{f + Z}. the set of all the measurable functions from X' to Z
PL
f#Px & (2) the pushforward distribution Px through measurable f E# the encoded data distribution
PZ
PAx
The conditional distribution Z given X, for forward Markov chain X Z The conditional distribution X given for reverse Markov chain X
QxiZ
Qz or
Simple and easy-to-sample distribution, typically a Gaussian
O
G#Qz or G# the generated data distribution
Qx
Ge#t, the generated distribution at diffusion time
pE(2(c).PE(( pE(=)
Probability densities for B and P respectively
(a(2|=),
Probability densities for G(Z) and Q% respectively
E:X FCF,
Encoder to map data point X ~ Px conditional distribution over Encoder hypothesis Generator to be learned that when applied to Qz. matches data distribution Px Generator hypothesis set
6:2-9(()
9 (
Px
Σ 8x,, the empirical measure with observations, where X; Px- Sample difference for the encoder-generator path.
x x
(E,G)
Ex-px ExwrBxxGE [Ao(X, Z, X)). generation error
(F,G)
E RZ-E(X.) (x & Ag X, Z, Xi), empirical reconstruction
(
Es C7, GE, 2x x G)), generalization gap for generation
T
Diffusion time length
Forward diffusion process satisfying dX, f)dt X(t)dW,, Xg ~ Px
(Xchelon
Ideal backward diffusion process satisfying (Xe) (X
(Xtheeoo.)
Approximated backward diffusion process with generator G., E [0,7]
Drift coefficient
E
Diffusion coefficient
(Wihejo
Wiener process/Brownian Brownian motion
17

---

Published as a conference paper at ICLR 2025
A.2 DEFINITIONS
Definition A.1 (f-divergence), Let P and Q be two probability measures defined on X with P < Q. Given a convex function f IR+ + R U {0o} with continuous extension at 0 and f(1)
define the f -iivergenee to be:
(PI()) del Ee [ (e)
As particular instantiations, choosing f(x) = x log x yields the Kullback-Leibler (KL) divergence DeL (PI)Q) and f(x)= (xr log (x + 1) log (²) yields the Sensen-Shannon (JS) divergence
(Ps(P]1)) (Q).
Definition A.2 (Fisher Divergence). Let Pa and Q be two probability measures defined on X, then, we have the fisher divergence:
Drisher (PI[Q) def Ex-r [IVV x logr(X) Vx log (
where p(z) and q(x) are the probability density functions.
Definition A.3 (Lipschitz function). Let (W, II) be normed space. We say function f f: : - R
is L-Lisschitz if for all w1, W2 W. (f f(w2)l L|[ w2ll-
Definition A.4 -aub-Gaussian).. Define the cumulant generating (unction(CGF) of random variable def
X as x (A) log E[c(X-E(X)) X is said be Suub-Gaussian if
R²R²
x S 2 V.ER.
Intuitively, sub-Gaussian random variable demonstrates exponential tail decay at rate comparable to Gaussian random variable. Positive number R is its analog for variance, often called the variance proxy. Entailing many common distributions, sub-Gause uub-Guusiianity is a standard assumption on the residuals in the analysis of ordinary least squares (OLS) and more recently, been widely used to provide non-vacuous bounds for deep learning algorithms Negrea et al. (201)).
Definition A.5 (Mutual Information). Let X and Y be arbitrary random variables, and DRL. denote the KL. divergence. The mutual information between X and Y is defined as:
I(X;Y) = ( [P)
Definition A.6 (Conditional Mutual Information). Let X, and Z be arbitrary random variables, The disintegrated mutual information between X and Y given Z is defined as:
1 (x; def (2
The corresponding conditional mutual information is defined as:
I(X;YIZ) EzIT"
Definition A.7 (Coupling). Let (X, p)) and (F,v) be two probability spaces. Coupling means constructing two random variables and Y on some probability space (2, T), such that X x > (projx) # p and (projy) # v, which means that и is the joint measure on X x y with marginals p. 1/ on X and y respectively. The couple (X,Y) Y') is called a coupling of v).
Definition A.8 (Wasserstein-p Distance). Let the two distributions be defined on the same Polish metric space (X, p), where :)
)) is metric and p E [1, +00), T(, v) is the set of all the couplings (see Definition A.7 of p.v. The Wasserstein distance with order p between and v is defined as:
Dw, def inf [xlll(p..) Xxx
/ plz, x 2' )dT(Tx)))
1/9
A.3 USEFUL LEMMAS
Lemma A.9 (Donsker- Varadhan Representation [Corollary 4.15 [Boucheron et al. 2013)). Let P and Q be two probability measures defined a set X. Let g : X + R be measurable function, and let Rexoleex (x)) & co. Then
DK (PII)) = (T (x())) log Enmolexp
g())) (()))
18

---

Published as a conference paper at ICLR 2025
Lemma A.10 (Decoupling Estimate Xu & Raginsky
(017)). Consider pair of random variables
X and Y with joint distribution Px,,, let X be an independent copy of X. and Y an independens
copy Y,, such that Px.y PxPy. . For arbitrary real-valued function X RR
R-sub-Gaussian Gaussian under Pxy. then:
() YOG (2RI(X;Y)
The above lemma has generalized extension in Bu et al. 2020), which directly assumes conditions on CGF and can cover other assumptions like sub-gamma.
Lemma A.11 (Girsonov Theorem, c.f. Theorem 8.6.6. Oksendal 2013)). If B, is an Itō process solves d.B, = alw,s))de + dB's for w E Ω 0 S S T, and Bo = O, where a(w,8) satisfies T exp 2 Jo a(w,s)ds) < oo for each w, then B, is a Brownian motion with respect to Q
where
dp (w) def exp (1o alw, a(awws))BB 2 2 |la(w, s)1Bds)
dQ
B GENERAL OPTIMIZATION OBJECTIVE AND TWO VIEWPOINTS OF DMs
The following is a regular derivation using basic probability knowledge. Similar results can be found in Kingma et al. (2019) with specific parametric notations in VAE. We now give a general version:
G(Z)Qz)
DRL s DKL (PxIG##222)) + Ex inf BEE (KL(E(X)
G#Qz
inf REE DxL(Px|IG#Q2) ExDxx(E(X)I G(Z)Qz)
G#Qz
/f |Ito) p(x) dr
inf EEE
40 (x)
/ p(x)pe(2(1)) log
pF(2|z)
((3)9c([1))/9((a)) dedr
/ p())pe(t())) log (())p((2(x) dzdr [[x](PP((X)]((())))))
inf EEE
(<))gc(z(z)
inf
EEE
B.1 DECOMPOSITION FOR VAES
The above general objective can be decomposed as:
inf [Dx)(()((()))(())))))) inf BEg
[x Bz-E(X) |- log (ogg)(X(2)) EEE
+Ex-P DKL (xxx))
x inf [E (EZ~E(X) - log (G (x (X (B((X))))) FEE
which is the common VAE objective, with the first term being the reconstruction loss and the second term being the distance of the approximated posterior to the predefined prior. Similar results exist in ratt-distortion theory, where the first term is interpreted as distortion, and the second is the rate.
B.2 DMS AS HIERARCIICAL VAES
Some previous work consider DMs as Hierarchical VAEs, such as Huang et al. (2021); Tzen & Raginsky (2019); Kingma et al. (2021). ). In this setting, assume that each encoder is conditional distribution on previous encoder's 's output and the initial input X = Xn ~ Px with E, (X, = Px, X. Similarly, the generator's output is conditional distribution previous generator's
19

---

Published as a conference paper at ICLR 2025
output with Cr-++1(Xi) = x Xy Z Qz. Using similar decomposing approach as VAEs gives the variational objective of diffusion models:
= (2 (2/0r- log p(z)pE, (1nl,)) (2/ dx...dz
ExwPy (PLLPEE (21x) + E (m (oggc, ... (x/(²1)
(
+ Px
B PE, (2) (AT (xe (2,, 22
(B+ - ((Px))
Ex~Px (DKL. # (X)))())))))) + log Qur (XIX1)l)
ExxPr
Max x x XIG
where PE, (2 = Pag (oe (p,, (m)
and the Markov assumption are used. The above objective is the same as Eq. (11) in Kingma et al. (021). The first term is the prior loss, the second term is the reconstruction loss, and the last term is the diffusion loss. This formulation is much more
PE, (nr 1/2)
complex for conducting a generalization analysis than the one introduced in the following. Hence, the theoretical results of diffusion models will focus the other.
B.3
DMs AS TIME- DEEEDEEPND MAPPINGS
The KL divergence between joint distributions can have the following decomposition:
inf [<][[XE(XXX]]]))22221 inf DKz EEE
EEE
where 2 is the reverse of E satisfying 2 Px dependent mappings, we have:
Ps Xi2
Considering DMs as time-
DKL(P (12))1(222))) - E 2~ 1 [[T
For X
DD (1
E
( Er
x
[[)
(EL (12) E B2 D KL
Q
Gur
where the last step holds by relaxing the last state measure inequality. Then, Then,Soon al. (2020b) upper bound the second term with the weighted score matching objective using Girsanov theorem. Hence, we have shown that VAEs and DMs are inherently optimizing the same objective.
path measure under data processing
C PROOF OF MAIN THEOREM
C.I GENERALIZATION BOUND FOR GENERATION (PROOF OF THEOREM 4.1)
Theorem C.1. For any encoder
E and generator G learned from the training data
(X+) assume that the loss ( X, is R-sub-Gaussin (See definition in Def. A.4
under P3,2x Qxiz X Qz X Px, where Z Qz N G(Z), X, Z are respective
independent copy of X and Z such that X, Z 11 X. Then, VX, S, Z4 ~ E(X,), & G(Z,), the generalization gap admits:
V2R Σ
gen Px C (E,G,T)|
Ex,
([X
20

---

Published as a conference paper at ICLR 2025
Proof. In this case, X, 2 ~ Qxiz x Qz. which satisfies X; 2 11. X.
For any 1) E I. let the cumulant generating function (CGF) be
-xx) 2xx))
x (1) log E 2,
1) x 2, A (X, 2, X)].
R. X. log
Let 2x,2,x be the joint distribution of X
Px, 7 ~ E G(Z). Using the Donsker-
Varadhan Representation in Lemma A.9 have bn R:
DD DKL xx,xx xx2xx) DRu P
x XxxxxXXX2xX/xx>>
-sup (xx2xx((X log E 2.2.x [001
g
2 1ExxxIA0 xxx - 7 x2x Ao(e, - xxxxxxxxx) (n).
(6)
In addition, the generation error admits
Lpx (E.G) = Ex-Px Bz B Axl
thus, we have:
EsLpx (F,G) ExxPx Ez Ey Bg x A (X, Z, X)]
= =Ex-rx E E xx x
Ax z, x)]
= x A 2, X)),
where the first equality holds because X is an independant copy of X. For the empirical reconstruction error
L G) E Ex B2
Ax 2x,
m
we have
1 m Es px (F,G) G) m
Σ
Ex,rx B Ao( Xi).
Hence, the generalization gap for generation error is Ao (E, = Es gen Px
cpx (E,G)
Cpp (E.G)
1
B ( A 2, X.)) - Bx.z..
Combining this with Eq.
O gives
m
Do (E, G.
-ngen Px
< m
(xx + (1)
12
Ex, Dx x
An R R
2
( + R²
2 An V.
21

---

Published as a conference paper at ICLR 2025
The last inequality is by the R-sub- Gaussian assumption and the last equality holds because the reconstruction process and the generation process use the same generator G. Dividing both sides by for 17 > 0, it gives:
Ex, Ex.DK
nR²
-gen dG (E, GT
vn 0,
2
since $/4 + by 2 2ab, Va, b 2 0, > 0, so we have
Ag (E, G n
V2R Ex. [1
-ge
Analogously, for 0, we have
gen DG (E) 1
([X;
-nTR?
.n < <0,
-1
2
i=1
where we have Px X,,ii., = -xx,,x
because E(X.), ~ G(Z), X, ~ Px. Hence, it gives
Ad gen (E, GT
1
VēRy Ex (XX
m
Finally, we get
V2R
Igen Ae (E,
VVE [Xx_)
71
Concludes the proof.
C.2 GENERATION ERROR BOUND (PROOF OF COROLLARY 4.2 AND 4.3
Corollary C.2. Under Theorem 4.1 let Ao(X, Z,X) || between the data distribution Px and the generated distribution G# is upper bounded by:
X|I. Then, the Wasserstein distance
V2R
Dw, (Px]( <
Ex +XX
m
Proof. By definition of Wasserstein distance, we have:
Dw, (Px|(GHT)
inf NxX ||I- 101-2'11(((2111
< ExaPxEx EX 11 -xlI
ExxPs Ezaz Ex x 11 -XI
-F
V2R m
Esspy (E, G)
Ex, (xxx)
m
The last inequality follows from Theorem
22

---

Published as a conference paper at ICLR 2025
Corollary C.3. Under Theorem 4.1 let the density function of probabilistic decoder G given latent code 2 be qG(-12) X + R3 and Ac(X, X)) log 9a (X The KL-divergence between the data distribution Px and the generated distribution G# is then upper bounded by:
V2R Σ
DRL = Es Cs G)
VEx.Dx + -x13
where h(Px)= = Ex1- log logp(X)) [[ denotes the entropy.
Proof. We have
p(x)
OKI(PxIIGHT) = p(x) log
(=)2)2 dr
= -h((xx)- / p(x) log (ʃ (f(3))a(o(2)d)) dx
S -n(x) / plz))(=) log (a(())))ddz
= =-M(Px) / p(z)a(t) / Ac(t, 2, x.)
=-h(Px) Bx AC 2.X)
=-h(Px) (
/2R
S Esspp (E,G)
Σ Ex, Ex, (E(()(] I(X;; A(Px).
m
The first inequality is from Jensen's inequality and the last incquality by Theorem 4.1
D DISCUSSION OF THE VAE BOUND
D.1 DETAILED COMPARISON TO PREVIOUS VAE BOUND
As discussed in Sec.6.5.2 Alquier et al. 2024). the mutual information bound is tighter than the PAC-Bayes bound expectation. For more concise form of mutual information bound, we transform the "Catoni-style PAC Bayes bound (Theorem 5.2) in Mbacke et al. (2023) to the expectation bound by integrating over the high-probability guarantee: Then, for any À > O, the following bound holds for any Ex(r) = N (x), diag(a3 (x))aa)) )) and fixed Go(z) -V (N)o(oz),
1
Dw, (Px s Es
Ez E (x. Ex -XIll
1=1
m
AA2 8m
Ke Es D
m
+ Es
(
Dw, (X.(())
m
i=1
where A = supo,a' 1/ x'll is the diameter of the bounded input space, Ko is the Lipchitz constant of μg. Since a + 2 Vab, Va, 2 0, we have
2x
A
Ow,(Px11Q7, & EsL (Fo,, Go) Go)
Ex, DKL (Fg(())))
2m
i=1
K
Ex. Dw, (E,(X))l
m7
Both the Wasserstein-2 distance and the KL-divergence control the generalization of the encoder. Specifically, since π = (O, I) Gaussian, we have Dw, (E(X,) (T) < (X.())l) according to the Transportation Cost Inequality. The above bound can be further formulated
1
Dw, (Px11Q%.) S EsLry (Fog, Go) + + V2KK)
Ex, Dxi. (Fg
23

---

Published as a conference paper at ICLR 2025
To be noted, the above bound only holds for specific Gø, not all Gø. In contrast, our bound holds for all Gg. ie., considers the generalization of the generator.
V2R
Dw, (PxIQQE.) & Ex (Es, Go) +
Ex (x
m
V2R
I Ex (Eg, Go) +
Ex [xx())
m
V2R
T(X;; X.1Z)
The bounded support assumption w.r.|] implies sub- Gaussia with R =
Duchi 2016). Hence,
we have:
1 m
Dw, (Px (Px)) S EsLpy (E.. Go)
E (Ex(((())))
A
I(X; X, Z2)
Ignoring the additional generalization term of generator, bound is tighter than previous work without the unnecessary Wasserstein-2 distance or could be considered has smaller factor without y2Kg. However, not all sub- Gauusaasn random variables are bounded, so our assumption more flexible and valid for unbounded support. As discussed in Sec.5, minimizing the first two terms in the bound is equivalent to the empirical 3-VAE objective:
[vΛE(0, Lp (Es, Go) + m Σ (x)))(())
where 3 is the regularization constant.
D.2 ESTIMATION OF THE CONDITIONAL MUTUAL INFORMATION TERM
To estimate the bound for VAE, the difficulty lies in estimating other two terms are easy to compute because Eg and Go are tractable distributions in VAEs.
I(X;; Xz|Z.), where the
To address this, we can bound the conditional mutual information term as
T(X: X(I%) <
Es
Ez (x
Es
which can be estimated by sampling several times m data points from the dataset, and then calculating the bound by replacing (Z,) as some data-free prior.
In the following, show how prove the above bound step by step. We first prove that
1 xxx,; < m
According to the chain rule mutual infomation, we have
1(x;: 1(x;; X1mm 11m) I(;;
Σ 188; X1I,, X13-1).
24

---

Published as a conference paper at ICLR 2025
Moreover, have
X1x-:;11 + (1
= - x|
Since both encoder and generator are learned from dataset S, we have the following Markov chains
X, + Zi X1:5 + 24; Z: + X; X1, X XG, Vi,ij [m]]
1X::j-1; X3 Zi, X:). which gives I(X;; X1:j 11j-11 (.) -
This can be derived with the mutual information chain rule:
(X;; (2) = X1j- Z.) (Xx
where (Xx; X1:--1111 X; (,) > 0. Therefore, we have 1 x 2 ((x;; and can further obtain
T(X;; T(X;; X X1:m Zz)
Σ 1( Z,) mI(X;; Vi e [m].
The last equality holds because learning G. with objective of VAE equally depends each datapoints, known as the symmetry in stability and generalization (Bousquet & Elisseeff] 2002 Bu et al. 2020).
The mutual information itself hard estimate since it's distribution dependent. We could the variational form by introducing an additional conditional distribution Gg(Z,), where 0 is some random initialization of the generator network. Then, we have the following:
p(2)) / p(0))|z) s|2) log
p( 2)
((122) didsdz
- (x DBB2::55AA,,,pDrr000002222111111 (raaaaaeeAA.aaaaaaoororaeee.aaaa E.BB,-Ex,,,Drr.GG22))))111,1
By selecting a duplicated decoder network with random initialization as reference, we can estimate the generalization of the generator Go with only the train data.
D.3 EXAMPLE ANALYSIS OF LINEAR VAE
We analyze simple linear VAE models following Ichikawa & Hukushima 2023 2024 Let Eg(z) N (Ha(2)), o2Ix)) with px(x) J
Raxxd' and Go(2) = (Nu Ia) with μ48 O² 2, 0 E Rd xd म = N(O, I,), x Rd Rd Combining the estimation for the conditional mutual information term, where duplicate randomly initialized generator Gz(z) = N(ug(=), Ia) is needed. Then, we have the following bound:
1 V2n (02 - 1)d' d' log(o2) + 7
0²
TTT xz
0
0112.
2dm
1=1
Consider the proportional limit setting with a terms: AB (-) Σ xt <ooTa.)) ax) for encoder and convergence analysis as future work.
z
(1), we have the following two major
0 2||2 for generator. We left the
2m²
25

---

Published as a conference paper at ICLR 2025
E PROOF FOR DIFFUSION MODELS
E.1 PROOF OF LEMMA 6.1
Lemma E.1. Let (Xi)T=o be the empirical version of the forward diffusson process defined in Eq. B). where Xo ~ Px. We assume the existence of the backward process under the regularity conditions outlined in Song et al. (2021) and denote it as {X.)r = (X,) which results from the reverse-fine SDE defined in Eq. (4) Then, the generative backward process {X) is defined in Eq. 5 Let Er, E GG,, Vt E [0,T] be their corresponding time-dependent Markov kernels. The density function of any generator G, given latent code z, is denoted qa((1z) X - RR. and let A Z, X)
log (X) Then, have
px (ET. GT) - Pxx [E ET 1 S _"
1
>²(t)D Fisher (Pxx 110 ddt..
Proof. From the definition of the empirical reconstruction loss, we have
Cpx (ET,ET E Xx Bx Ko (Xx) E (- lol1og 1oggum7 (Xol (oo(X))
(xx) A (Xo, XT, Xo)
Ex
= =xxo Ex ExywEr(No) (- (-log loog 9 (X (XolXr))
Analogously, we also have
Cpx (ET, GT) = E Ex (x) (Xo, Xr, Xo)
ExpxPy (x E Xo-Gr(X)) (-log 90, (Xol (x))
= =xx, Ex Ex)~Er(Xn) (- laz 4a, XT))
These two empirical reconstruction losses aim to compare the corresponding SDEs, both of which start from random draw from the aggregate posterior induced by the encoder.
The first the backward process given the empirical data distribution, characterized by E, g,1 e [0, T]:
dx, = (f(x,))),))) X()) + (()dWW Xy M(),,,
Pxr.
The second is the generating process, characterized by Gr, [0. 7):
ax, = () - (lg ( - A(I)dW, &r ~ Pxr-
Therefore,
9g,1 (XolXT)
12 (ET, GT) Lpp (ET, (ET
E 3xnnAy Ex, log
9G, (Xo Xol)T)
!E (ro (xr)
E ExpxXx 90 (2ol log
Xx)
dro
(2ol
Expxxxy DRI(Q F
XolX, 10
11Q
26

---

Published as a conference paper at ICLR 2025
which gives
px (Er, (Er,Gr) Lx (Er; Err')]
< E2 PxprPx
DxL (9 (11T)) (117)) (1xx,)
Foj
E =xx Ee (0
\(t((Vlo)((X))) - V (X)))
/ x ((())\Voog&((x.) V log (x))
1=
E Xr Pxr
Eq BT
(1)))
( V log 2e (x:))lal
1
Ex (AP(I) (x,)) - V log a1 (x)
2 1=0
1 2 (1I (Px, 1x.))t.
The first inequality is obtained by applying data processing inequality to the Markov chain, where the KL divergence between the last iterate conditionals is smaller than that of the whole path, similar to the proof of Theorem in Song et al. (2020b). The subsequent equalities are from the Girsanov Theorem (Theorem 8.6.6 Oksendal (203)) and the definition of the Fisher divergence.
27

---

Published as a conference paper at ICLR 2025
E.2 PROOF OF THEOREM 6.2
Theorem E.2. Under Lemma 6.1 for any SDE encoder Et and generator G., trained via score match- ing on S = ) the corresponding output at the diffusion time T are Xr Er(X;), Xo ~ (Xr) for each X; S. The Ki-divergence between the original data distribution Px and the generated data distribution G9- # at diffusion time T is then upper bounded by:
DKL(Px|G9 S Es
DKL ( ESM (0, (()
/2R m
/2 I(xoi (1187)
Ex, (XL(
m7
m
Proof. At first, we combine the results of Corollary and Lemma 6.1 and obtain:
DKL (RI(Px|( s Es
(Cpx (ET,
& (0, (0,
+ V2R
/ D (xx))))(( ((x,))) h(Px)
Eg
px (Frr, (Frr, LESN LESM (0)) >((x))
+ 22
V2
Ex. (E
==1
Then, the reconstruction error Lrx (Er, ET of the reverse SDE can be decomposed as:
pp (Er, ET
E x Ex (xx) log (Xolxx))
pr (Xr)
= Px Ex Ex log
Pey (X X ) (Xo)
m D (X)
= =h(Px)=
m
which will converge / h(Px) when 7 - oo, because of #x any data distribution and h(Px)
P-'#Pxx Px. So 7 oc and m - both hold, we have EsCp, (Er, -
Considering the normal case, we have
Es [s(h(Px)] h(Px) 00
because the entropy function h((p) Jensen's 's inequality (See Verdú (219)))
p log is concave, and we can take the expectation inside by
Therefore, we have:
m (KL(E) + +x(x))
EsCpy Py (Er, ET,, < Es
Combine all these, we conclude the proof.
28

---

Published as a conference paper at ICLR 2025
F PROOF OF THEOREM 6.3
Theorem F.1. Let the step size be T = f. where we split T to N discrete times. For any ls E [N] we use the following discrete update for the backward SDE by setting EI,, NO (s), T Tk:
X12 - -1- - -T- ( - kk-1)
VTN(T 1k-1 1112)1
Furthermore, we assume a bounded score V log (c)) I L, Vx,t. Then, we have the
m
I(No; X1am (1 TL2 " X2 1T
m Σ X((x))
= m
2mN
Proof. At first, let us denote the sequence of generated data as [IM def [Xe,y... Xewle then, we have the following Markov chain:
+8
Xo
+
t
Xx
According to the mutual information chain rule, we have:
X(xo; 1 X1 (8)) + Σ El( X.18T X 11--1) 18oo
(XT) = 1 X X, XT) that can also be decom- posed I(Xo; X₂) (T) + (( X,|X7, Xo) and I(X;; (1=-1(X7) the last inequality holds with 1(x1:1=;; X3 1r Xo) 0.
Since 1 1(xo: X, X 1
For any k E [N], we use the following discrete update for the backward SDE, where ct. (0 Ia), 4 = T - Tk, T= T
N.
X, (1 2 >²((T to Xmm - -t-11) T -x-1) VTA(T -
Since the approximation sy (X tk-1) Vx.,,, log Logrr-u-1 (X ) is determined by S = Xim under some functional form, simply denote it as gs (Xt..)) we can consider the above update as a Langevin dynamics
X12 =11- (1- 2 _x²(T to 11)1)) + 1X + 0k-1ft--11
where 77-11 = TA2(T - tx-1) = o Then, we can apply the technique in Pensia et al. (018).. and obtain:
N
I(Xo: X1.m Xx) & I(FIM) X1:m (2)) S Σ
I I(X : X1:m Xx, X &b1-11
k=1
X - h h(Xes (1 Xr, xlt-ll))
According to the Langevin dynamic update and the bounded gradient assumption, we have
h(8 11)- h(n=-19s(X,,.)) + ok 1k-16)_1) S d log 2 I I.2 + do
2
where we use the fact that the Gaussian distribution has the largest entropy with h(Y) < $ log for all random variables Y satisfying B|| 2 < C. Moreover, we have
d
E 119 (Xee1) 0k- 16t 2 El 19 X x, >12 11lo
11 < 1/ 1.2 + do 2
29

---

Published as a conference paper at ICLR 2025
which is due the independence between the score estimation and the injected noise. Then, also have 1 1xx,,, X1m, Xr, X X1k-11)= (( I á log (2TreT
Combine all these, we can get:
I(xo: X1mm XT)
N
log
+
The L do
N
Th
L
2
20 2
1=1
1=1
where the last inequality use log(1 + x) S Vr 2 0. Putting 7]k-1 TA2 tk-1) = 0 the above equation, we conclude the proof.
= into
30

---

Published as a conference paper at ICLR 2025
G EXPERIMENT DETAILS
In this section, we provide the detailed experimental setting and some additional experimental results. Our experimental code is available at pllttps:: /ttp::///ithbb wwwmmoiiiieeeeeee...e InfoGenAnalysia The implementation is based on the code in Van den Burg & Williams 2021 for VAÉs and based on the code in Huang et (202T) for diffusion models.
Computational Resource The experiments for Swill Roll data were running on a machine with 2080Ti GPU of I IIGB memory. The experiments for MNIST and CIFARIO were running on several server nodes with 6 CPUs and GPU of 32GB memory.
G.1 VAE
To verify our theoretical results in Theorem 5.1 and compare to previous PAC Bayes bound for VAEs, present experiments on MNIST in this section.
G.1.1
RESULTS
test loss
Estimal ttdd Mobarl infommation
our bound
23.20
previous bound
Mem orizaaioon Score
Estimated Bound
190
20.13
w
125
memorization score 0.95)
10-
mutual Information
20
46
20
120
Epochs
Epochs
(a)
(b)
Figure 4: (a)The test VAE loss, previous PAC Bayes bound (converted expectation bound) and mutual information bound on MNIST dataset change over training epochs. (b) The memorization score with 0.95 quantile and mutual information estimation change training epochs.
We conducted experiments on the MNIST dataset, using a Bernoulli distribution as the generator Go for the VAE. The mutual information term estimated using the method proposed in Sec. D..2 leveraging a randomly initialized duplicate generator, which remained fixed as reference throughout the process.
The estimated bounds and test loss are plotted in Fig (a). The previous PAC Bayes bound estimated with train data goes below the test loss. This is due the bound does not hold for any Gg. In contrast, our bound estimated on train data well aligns with the test loss because we have considered the generalization term for Go-
31

---

Published as a conference paper at ICLR 2025
To further illustrate the effectiveness of using the conditional mutual information term capture the generalization of Gg. We compare its estimation to the memorization score proposed in Van den Burg & Williams (202T) which measures how much more likely an observation is when it is included in the training set than when it is not. The result is presented in (i), where we plot the 95% quantile of the memorization score and the conditional mutual information term along the training epochs. The two terms are highly correlated with similar evolving trends, suggesting our bond may capture the memorization some extent. To be noted, evaluating the memorization score requires an additional validation set, while our bound can be evaluated with only the training set.
G.1.2 EXPERIMENT SETTINGS
This section strictly follows the setting in Van den Burg & Williams (2021)
Data and network structure During training on MNIST, we dynamically binarize the images by treating each grayscale pixel value as the parameter of an independent Bernoulli random variable, following standard practice. The encoder block is the stack of FC(1024, 512), RELU, FC(512, 256), RELU and FC(256. 16). The generator block is the stack of FC(16, 256), RELU, FC(256, 512), RELU, FC(512, 1024), and Sigmoid.
Training Details We optimized the model parameters using the Adam optimizer with learning rate of 1) = 10-3 The training was conducted with a batch size of 64, while the remaining Adam hyperparameters were kept at their default values in PyTorch. The model was trained for 100 epochs.
G.2 DIFFUSION MODEL
G.2.1 BOUND ESTIMATION
Our main objective is to verify Theorem 6.2 for diffusion models, which involves illustrating: 1. the inequality holds, as illustrated in Fig. 2 the evolving trend of the two sides follows the change of sample size m, seen in Fig. 2 (a). 3. the trade-off on diffusion time T, showed in Fig. (b). To make such a comparison, we need a quantitative estimation of the two sides.
Recall that D 1Q measures the proximity of the original data distribution the generated data distribution. A similar metric used to evaluate the performance of generative models is the Fréchet inception distance (FID), which is the Wasserstein-2 distance between the generated and the original data distribution. Since the data distribution is unknown, we conduct a Monte Carlo estimation E log(p((( Xx)//G., (X.)) using test dataset of size m. = 1000, which is independent of the training set with Stte = {X), 1, X; ~ Px. Then, we use Kernel Density Estimation (KDE) to calculate both pf p(X;) and qG, (X,). Such estimation of QG, is disentangled from the diffusion process itself and can better reflect the generalization of the learned diffusion model by only using the generated data (one can sample any number of data wanted to fit the KDE).
On the Right-Hand Side (RHS). T2 has an analytic form. T3 is upper bounded by Theorem 6.3 where we use step-wise estimation for the maximum score norm similar to the gradient norm estimation in the literature of information- citemation-thhortticc learning (Pensia et al. 2018 Li et al.) 2019 Negrea et al. 2019). Ti is the KL divergence between the final time posterior and the aggregated posterior, where the former is multivariate Gaussian, and the latter is mixture of Gaussian in the normal setting of diffusion models. Thus, we can simply use KDE with a Gaussian kernel and bandwidth fixed to time-specific variances of the forward process to approximate the aggregated posterior. Combining the empirical score matching loss, we have the estimation of RHS, W.r. the expectation over S, we conduct 5-times Monte- Carlo estimation by randomly generating train datasets with different random seeds.
32

---

Published as a conference paper at ICLR 2025
G.2.2 SWISS ROLL
Experimental Setting
Score matching model structure We use 4-layer Multilayer Perceptrons (MLPs) with hidden size 128 to approximate the score function, where the input dimension is the 2D data dimension plus the ID time dimension.
Train-test details During experiments, we record the generated data and estimate their KL divergence from the test data during the training dynamics. We use 1000 Monte Carlo sampling for the test-data KL divergence estimation and the same sampling size for every kernel density estimation. The score matching model so (z, t) trained for 10000 iterations, and the backward generation takes 1000 steps, i.e., N = 1000.
Additional Results In Fig. and Fig. 6 we plot the generated data with the score model obtained the last iteration for each specific setting, e.g,, different train sample size mn and diffusion time T.
== 10
m= 100
m == = 200
m
m = 800
m =000
Figure 5: Sampling results w.r.t. different train data size m: 1000 data points generated by score- based model trained with 10000 gradient iterations and diffusion time T 1. The sampling is conducted after 1000 steps when solving the discretized backward SDE.
33

---

Published as a conference paper at ICLR 2025
T 0.020
T= 0.040
T= 0.060
T= 0.080
T= 0.100
T= 0.120
T= 0.140
T= 0.160
T= 0.180
T= 0.200
T= 0.400
T= 0.600
T 0.800
T= 1.000
T= 1.200
T 1.400
T= 1.600
= 1.800
Figure 6: Sampling results w.r.t. different diffusion time T: 1000 data points generated by score- based model trained on m = 200 data points with 10000 gradient iterations.
34

---

Published as a conference paper at ICLR 2025
G.2.3 MNIST AND CIFARIO
Experimental Setting
Score matching model structure Following Ho al. (2020) and andna al. (202I), we use modified UNets based on a Wide ResNet for 1x28x28 images in MNIST data and 3x32x32 images in CIFARIO data, respectively. The weight normalization is replaced with group normalization. Each model consists of two convolutional residual blocks per resolution level, along with self-attention blocks at the 16x16 resolution between the convolutional layers. The diffusion time / is incorporated into each residual block via the Transformer sinusoidal positional embedding.
Train-test details During experiments, we record the generated data and estimate their KL divergence from the test data during the training dynamics. We use 100 Monte Carlo sampling for test-data KL estimation and the same sampling size for every kernel density estimation. The score matching model 80 (z,, i) is trained was trained on m = 16 images for 10000 iterations, and the backward generation takes 1000 steps, i.e., N 1000. BPD was estimated using the method proposed in Huang et al. (2021), and test data KL was estimated using KDE for the Swiss Roll data.
Additional Results Large step sizes will cause instability large discretization errors. However, the step size is not the smaller, the better. After some threshold, reducing the step size further yields negligible improvements because of the model's s approximation error and will cause heavy computation burden. In addition, according to the relation N = T small step size corresponds to large number of steps, which can lead to overfitting, especially when the model is trained with few data.
Replace N with T/T in the bound, we have T3
Ry (Bt 5o
180 -0-------111 - In the
experimental setting of the original submission Fig 7 (a), set N 1000, T [0.2, 2). so have 0.0001 < r S 0.002. Since we used fu = 0.1, 31 = 20 in all the experiments, which gives 0.06 (1 + 8) TB, < 0.09998. Therefore, we have T3 E O(T) that has linear growth w.r.t T for all 7 used in our experiments. Hence, we suppose the impact of in this range is minor. To further verify this, we t 0.001 suggested by the reviewer and change N for different T accordingly. In 7 we compare the results with the previous setting. It shows the whole upper bound (including score matching loss), and the log density remains consistent with the results in the previous setting. However, we keep using 0.001 for the rest of the experiments to avoid potential concerns because we are varying T in a larger range.
bpd
bpd
tast data KL
upper bound
bound 228
25-
E2t Estimated Bound
200
Edtimated Bound
BPD
BPD
035
3.00
0.5
106
1.5
1.5
120
6.6
075
220
1.66 230 1.5
200
Diffusion Time:
Diflusion
(a)
(b)
Figure 7: The evolution of the estimated bounds, test data KI. divergences, and test data log densities (measured by BPD) w.r.t different diffusion time 7 for DM trained on few-shot MNIST data (m 16): (a) with fixed number of steps N = 1000 (KL and BDP were calculated with 100 test samples.) and (b) with fixed step size 0.001, BDP was calculated with 10000 test samples, note N = I.
35

---

Published as a conference paper at ICLR 2025
7
6
8
O
2 0
6
8 8 8
3
9
70
5 6
0
7
6 3 o
/
8
3
6
9
9
7
0
6 6 6 3
6 0 7
4
6
0
4
9
6
6
(a) train data
(b) T' = 0.2
(C) T = 0.4
(d) T = 0.6
(e) T
0.8
(1) T = 1.0
/
2 3 3
1
7
g g
4
4 6
8
Y
4 3 0
7
6
8
5
4
4
2
8
P
(g) train data
(h) T 1.2
-T=
() 1.6
1.8
() T 2.0
Figure 8: The generated images for different diffusion times T on the MNIST dataset (we randomly sample 16 images for each 7). The score-matching model was trained on m = 16 images (we use this few-shot setting to make sure we can present visual difference within limited random draws) randomly sampled from the dataset for each T. The training process takes 10000 iterations. The generation quality is consistent with the estimated bound in Fig. (a), where the optimal diffusion time should be around T 0.8. The sampling process has a fiexed step size r 0.001.
6
7
8
5
21
4
9 2
q
0
8
6
7
4
3
9
70
7
GAY
2
/ 2
1o 2
5
9
5
3
6
9 2
L
3
/
6
8
C
6
1
6 6
4
6
0
4
9
2
3
3
O
5
3
(a) train data
(b) 7' 0.2
(c) T' 0.4
(d) T
0.6
(c)
= 0.8
(1) 7 = 1.0
| 2 3 3
3 O
4
4
8
2
6 9
0
7 6
8
4
2426 6
2 3 0
2
(g) train data
(h) T' = 1.2
T = 1.4
(T = 1.6
(k) T = 1.8
(1) 1' = 2.0
Figure 9: The generated images for different diffusion times on the MNIST dataset (we randomly 16 images (we use
sample 16 images for each 7). The score-matching model was trained this few-shot setting to make sure we can present visual difference within limited random draws) randomly sampled from the dataset for each T. The training process takes 10000 iterations. The generation quality is consistent with the estimated bound in Fig. (a), where the optimal diffusion time should be around T 0.8. The sampling process has fixed number of steps N 1000.
36

---

Published as a conference paper at ICLR 2025
(a) train data
(b) T = 0.2
(c) T = 0.4
(d) 7 = 0.6
(e) 0.8
(1) 7 = 1.0
(g) train data
(h) T 1.2
①T 1.6
(k) T 1.8
()) T 2.0
Figure 10: The generated images for different diffusion times T on the CIFAR 10 dataset (we randomly sample images for each). The score-matching model was trained on m this few-shot setting to make sure we can present visual difference within limited random draws) randomly sampled from the dataset for each T. The training process takes 10000 iterations. The generation quality is consistent with the estimated bound in Fig. (b), where the optimal diffusion time should be around T 0.8. The sampling process has a fixed number of steps N = 1000.
images (we use
H BROADER IMPACTS AND LIMITATIONS
Broader Impacts
Potential positive impacts We study the theoretical aspects of generative models. By improving the generalization, we can decrease replicated generation help address the privacy and copyright issues in generative models.
Potential negative impacts The improvement for generating diverse data may be used generate harmful information or fake news.
How to address the potential negative impacts? We can design harmful information detection mechanisms and embed a filter strategy for generative models.
The theoretical analysis for DM in the last theorem is for a first-order Euler-Maruyama solver for SDE. One can extend and prove guarantees for other backward SDEs. While some diffusion models use deterministic encoders or generators that also work well in practice (e.g., [Song et al. (2020a) and Bansal et al.) (2024)), we consider the encoder and generator as randomized mappings as is most typical in diffusion models. However, our current definition of encoder and generator with randomized mapping covers the deterministic mapping setting by restricting F(X) and G(Z) the set of delta distributions. The problem is due to the mutual information terms could be infinite for deterministic settings. However, this could be addressed by exploiting other refined information- theoretic tools. As this paper focuses providing unified theoretical viewpoint for typical VAEs and DMs, we cannot cover all the methods. The improvements mentioned above will be left as future work. Moreover, potential faimess issues raised in learning generative models could be considered by exploiting previous related works like Xu et al. (2024); Shui et al. (2022).
Limitations
I ADDITIONAL RELATED WORKS
Algorithms for VAES The VAE (Kingma & Welling 2013) has been widely applied and improved algorithmically through numerous extensions that include changing the posterior distribution to exponential families (Shi et al. 2020; Shekhovtsov et al. 2021) and docation-scall families (Park)
let al. 2019), balancing the rate-disted distoristottion trade-off (Higgins al.] [017]; Rybkin et al.] (222)) replacing the regularization term with adversarial objectives (Makhzani et al. 2015), or using other divergences like the Wasserstein distance (Tolstikhin et al. 2017) (WAF).
37

---

Published as a conference paper at ICLR 2025
Score-based diffusion models Song et al. (2020b) unifies the previous two main diffusion ap-
proaches: Score matching with Langevin dynamics (SMLD) (Song & Ermon 2019) and Diffusion probabilistic modeling (DDPM) (Sohl-Dickstein al. 2015 Ho al. (020) score-based diffusion models, where their forward processes are considered as different families Stochastic Differential Equations (SDEs). Later on, the variational perspective of these models was studied in (Huang et al.) 2021 Kingma et al. 2021 Franzese et al., 2023)) Huang et al. (2021); Song et al. (2020b) study how use difflusion models to estimate the data likelihood based some theoretical results in stochastic calculus (Karatzas & Shreve 2014; OOsendaal 2013). Recently, latent diffusion models (Vahdat
al. 2021] Rombach et al. 2022) have gained great success in generating high-resolution images.
extended to further applications like text-to-image editing (Han et al. 2024 Huberman-Spiegelle
al. 2024).
Convergence theory for diffusion models De Bortoli et al. (2021) are the first to give quantitative convergence results for DMs, where they upper bound the original and generated data distribution in Total Variation (TV) distance and assume acccaucua score estimation. This leads to vacuous results under the manifold assumption, where the TV can be very large, even if the distributions are similar. Lee al. (023); : Chen et al. (2022) that also bound in TV but assume T,a-accurate score estimation have the same problem. Chen et al. (2023a) provide an improved analysis with minimal smoothness assumptions, which is valid for any data distribution with second-order moment with I acccurreeeee score estimation. The above works focus on analysis for convergence w.r.t population data distribution without explicit consideration of generalization.
cotormatiiontthoreee learning theory Recent information -twhooetttccc analyses Xu & Raginsky 2017, Russo & Zou 2019 Steinke & Zakynthinou, 2020] Haghifam et al., 2021 2022; Hellström & Durisi 2022 Wang & Mao, (023) have provided rigorous framework for understanding the deep learning models, and have further been extended to complex learning scenarios, such meta-learning (Chen et al. [021 2023b) and domain adaptation (Wul ct al. 2022, Chen & Marchand] (023). Unlike conventional VC-dimension and uniform stability bounds, this approach offers a key advantage in capturing dependencies on the data distribution, hypothesis space, and learning algorithm.
generalization capabilities
38


Reviews:
[{"review_id": "XxCClfOBss", "reviewer": "Reviewer_7s19", "summary": "This paper provides a information-theoretic framework of generalization theory for variational auto-encoders and diffusion models. Authors consider generalization properities for both the encoder and the generator in VAEs. Their generalization bounds can be estimated using only training data, providing a practical guidance for hyperparameter selection.", "questions": "- The generalization bounds resemble Theorem 4.1, which primarily links encoder mappings' complexity to the gap between empirical generalization and expected generalization. But this should also affects empirical generalization. Can you provide a theorem telling how this two terms together affects expected generalization? \n- As far as I know, in practical works of generative models, the target distribution is always unknown, which means the empirical generalization error is always unknown. Could the authors elaborate on the practical benefits of bounding the gap between expected and empirical generalization error, particularly when the empirical error itself may not be measurable? \n- As mentioned in line 350, generalization bounds are computable, can you explain how to compute term $\\hat{L}_{ESM}$ among upper bounds in Theorem 6.2?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors address the critical topic of generalization in generative models, and provide estimable bounds for both the encoder and the generator in VAEs. \n- Bounds for VAEs avoid Wasserstein distance and impose milder assumptions (bounded to sub-Gaussian). \n- Bounds for DMs overcome the challenges associated with KL-divergence's non-satisfaction of the triangle inequality, and contribute to a clearer understanding of diffusion time’s role in generalization and model performance.", "weaknesses": "- In line 98, the paper asserts that the bounds for the encoder are tighter, yet this claim lacks sufficient detail. Although some comparisons to previous bounds are made in line 324, there remains a need for a more explicit, quantitative analysis to illustrate the improvements over existing bounds. Adding a direct comparison or detailed quantitative analysis would make the claim more substantiated and provide clearer evidence of the improvement.\n- The proposed generalization bounds do not clearly indicate a dependency on the number of samples, $m$, limiting their practical applicability. To make the bounds more actionable, it would be helpful if the authors explicitly stated the sample complexity (e.g., $O(1/ m)$) within the main theorems or discussed the bounds’ scaling with respect to $m$. This addition could guide readers in understanding the bounds’ robustness and sample efficiency.\n- The paper suggests a theoretical trade-off for diffusion models based on diffusion time, but this trade-off is not consistently reflected in the experimental results, as KL and BPD metrics do not show this effect. The authors could address this discrepancy by either providing a potential explanation for the divergence between the theoretical and experimental findings or suggesting additional experiments that might better capture the trade-off. Revisiting this section would enhance clarity and ensure its alignment with the paper’s core contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rPLNfZWvSU", "reviewer": "Reviewer_qVWA", "summary": "This paper derives generalization bounds for encoder-generator-based generative models, considering both the encoder and generator components. The analysis is specifically applied to Variational Autoencoders (VAEs) and Diffusion Models (DMs), with tailored generalization bounds presented for each. Notably, a novel trade-off relationship is introduced for DMs. The theoretical results are supported by numerical experiments, demonstrating that the derived bounds offer practical utility, particularly in capturing the trade-offs in generalization.", "questions": "- The paper recommends adding terms for the generalization of $G$ to provide insights and practical guidance for VAEs. However, estimating these terms is generally considered difficult. Are there any methodologies that could facilitate this estimation?\n- The few-shot setting with $m=16$ is mentioned in the experiments on real data. Could you provide more details on this experimental setup? Does it involve generating new data with a pre-trained model in a few-shot learning manner, or is it simply trained with limited data?\n- What is the definition of $g$ in line 209?\n- How would the generalization bounds change if the data were assumed to lie on a low dimensional manifold? What predictions can be made regarding such cases?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper derives a generalization bound for encoder-generator architectures under the relatively mild assumption of sub-Gaussian loss functions. As noted in lines 286-291, the paper provides an intuitive explanation of these bounds and a convincing discussion of the trade-offs involved.\n- Corollaries 4.2 and 4.3 extend the analysis to evaluate the Wasserstein distance and KL divergence between the generative model's distribution and the data distribution, offering valuable tools for the theoretical analysis of a wide range of generative models.\n- For VAEs, the paper presents a bound that applies to arbitrary generators, distinguishing it from existing research.\n- In the case of DMs, a new trade-off relationship is derived. Under certain reasonable assumptions, the generalization bound involving mutual information of $G^{θ}$ is established concerning sample size, score function bounds, and time $T$. This leads to a tighter $ m^{-1/2}$ bound.\n- The results are further substantiated by numerical experiments that validate the theoretical trade-offs observed in the upper bound.", "weaknesses": "While the results are significant in terms of learning theory by considering the effects of both the encoder and generator, some areas could be further improved:\n- Although challenging, the analysis does not incorporate the complexity of the learning models. Including bounds related to the complexity of simple neural networks or linear models could strengthen the work.\n- Aside from the theoretical analysis provided by the generalization bound, it would be beneficial to relate these results to those from sharp theoretical analyses in proportional limit settings. For example, in denoising encoders [1], autoencoders [2], or variational autoencoders [3, 4], the generalization error has been sharply evaluated by fixing the ratio $\\alpha = n/d = \\Theta(1)$, where $n \\to \\infty$ is the number of data points and $d \\to \\infty$ is the data dimension. Including such discussions in the related work section would provide a more comprehensive view of the literature.\n- It would be interesting to see the results of training on the complete datasets of MNIST and CIFAR-10 rather than in a few-shot setting.\n- The paper refers to the claime by Thesis et al. (2015) that \"accurately estimating the KL divergence and Bits-Per-Dimension (BPD) for high-dimensional data distributions with limited data is challenging\". However, this could be addressed using population MCMC or other Markov chain Monte Carlo methods. Investigating whether the trade-offs as in the upper bound remain evident in such scenarios would be intriguing.\n\n- [1]: Cui, Hugo, and Lenka Zdeborová. \"High-dimensional asymptotics of denoising autoencoders.\" Advances in Neural Information Processing Systems 36 (2023): 11850-11890.\n- [2]: Refinetti, Maria, and Sebastian Goldt. \"The dynamics of representation learning in shallow, non-linear autoencoders.\" International Conference on Machine Learning. PMLR, 2022.\n- [3]: Ichikawa, Yuma, and Koji Hukushima. \"Dataset size dependence of rate-distortion curve and threshold of posterior collapse in linear vae.\" arXiv preprint arXiv:2309.07663 (2023).\n- [4]: Ichikawa, Yuma, and Koji Hukushima. \"Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t9tqOt8Rzr", "reviewer": "Reviewer_huQG", "summary": "In this paper, the authors explore the theoretical generalization behaviour for both variational autoencoders (VAEs) and diffusion models (DMs), noting that a DM can be seen as an infinite-layered hierarchical VAE. For both models, the behaviour of both the encoder and decoder in the VAE, and forward and reverse diffusion steps in the DM are analyzed. The authors then provide a theoretical upper bound for DMs with respect to the number of diffusion steps and then empirically verify it on several datasets, both synthetic and real (MNIST and CIFAR10), demonstrating its validity and usefulness in training optimal DMs.", "questions": "I suggest the authors make the writing more clear by including a separate background section on the relationship between VAEs and diffusion models; this will make it clear that the main contribution is providing a theoretical upper-bound for diffusion models as aided by the analysis of VAE generalization. A good place to start is the Variational Diffusion Models paper [1] that explicitly makes this connection by formulating the diffusion learning algorithm in terms of a variational lower bound.\n\nReferences\n[1] Diederik P. Kingma, Tim Salimans, Ben Poole, & Jonathan Ho. (2023). Variational Diffusion Models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper provides a very detailed and comprehensive information-theoretic analysis of generalization in VAEs and DMs along with experiments that empirically validate it. In particular, the incorporation of encoder-decoder / forward-reverse process into the analysis provides a novel view into their impact on the generative models' generalization behaviour, such as the finding that longer diffusion steps do not necessarily result in better estimates in DMs.", "weaknesses": "The paper's writing made it difficult to process the main contributions to the paper for two main reasons:\n\n(1) Despite the abstract suggesting that the VAE's generalization behaviour is studied, much of the paper's focus is on analyzing DM behaviour.\n\n(2) There is notably no experiments that validate VAE behaviour, which suggests that the VAE is studied here as a precursor to understanding the generalization behaviour of DMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s0ptqC1UHW", "reviewer": "Reviewer_o393", "summary": "This work puts a bound on the generalization gap for generative models relying on an encoder-decoder architecture. \nUsing this, they introduce bounds on \n - the wasserstein distance between the data distribution and generated distribution for variational autoencoders (VAEs),\n - the KL-divergence between the data distribution and generated distribution for diffusion models. \n\nThey run experiments on a score based diffusion model using data from \"Swiss Roll\", MNIST and CIFAR-10.", "questions": "Q1: Do you have a theoretical reason (or at least an intuition) for why the mutual information should follow \nthe upper bound in theorem 6.3?\n\nQ2: At the end of section 6.1, you write \"$T_3$, which characterizes the generalization of generator $G_T$ , will\nremain non-zero for a small sample size\". How small does the sample size need to be? And if you use enough samples in your \nMNIST and CIFAR-10 experiments to get a good estimation of KL-Divergence, will that make the contribution of $T_3$\nvery small? \n\nQ3: Could you add the next steps in your proof in appendix 1.C? As it is now, I don't see the final claim of the theorem \nin the last lines of the proof.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1: VAEs and diffusion models are widely used, so theoretical work on them is very relevant. \n\nS2: The bounds introduced in the work are new. \n\nS3: The story of the article is easy to follow.", "weaknesses": "The weaknesses relate to the experimental part of the paper.\n\nW1: For both experiments you are using a fixed number of steps for generation while changing T. \n\tThis means that for varying T, you are also varying the step-size, which can have a large effect \n\ton generation (See e.g. Zhang and Chen 2023 \"Fast Sampling of Diffusion Models with Exponential Integrator\").\n\tIt is not clear to me whether this also affects your experiment in other ways (you should check this),\n\tbut if you want to say something about sample quality, my suggested solution is to choose a step size, e.g. 1/1000,\n\tand also generate images with varying T, but fixed step-size. \n\nW2: Figure 3 b) and c) do not support your claim. You say that the discrepancy comes from using too few samples \n\tto get a reliable estimation of KL divergence. My suggested solution is to do the experiment with enough samples \n\tto get a reliable estimation. Since your result is about KL-Divergence and not sample quality, it is not enough to \n\tlook at sample quality for verification. Especially since since performance in one is not directly linked to \n\tperformance in the other. This observation is also made in Theis et al. 2015 \"A note on the evaluation of generative models\"\n\twhich you also cite yourselves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)