Theoretical guarantees for Deep Generative Models:A PAC-Bayesian Approach
Sokhna Diarra Mbacke
Université Lavalsokhna-diarra.mbacke.1@ulaval.ca

March 2024

S. Diarra Mbacke (ULaval)

March 2024

1 / 44

Co-authors

S. Diarra Mbacke (ULaval)

March 2024

2 / 44

Summary
1

Context2

Preliminary Concepts
Generative models
PAC-Bayesian Theory

3

Results for GANs
Bound for Wasserstein GANs
Numerical Experiments

4

Results for VAEs
Variational Autoencoders
Bound for the Generative Model

5

Conclusion

S. Diarra Mbacke (ULaval)

March 2024

3 / 44

ContextGenerative models are widely used.
Determining if a generative model generalizes well is a difficult
problem.
PAC-Bayes is a powerful tool in statistical learning theory.

Goal: Use PAC-Bayes to study the properties of generative models.

S. Diarra Mbacke (ULaval)

March 2024

4 / 44

Next up
1

Context2

Preliminary Concepts
Generative models
PAC-Bayesian Theory

3

Results for GANs
Bound for Wasserstein GANs
Numerical Experiments

4

Results for VAEs
Variational Autoencoders
Bound for the Generative Model

5

Conclusion

S. Diarra Mbacke (ULaval)

March 2024

5 / 44

Generative Modelling
Given finite iid samples:

S. Diarra Mbacke (ULaval)

March 2024

6 / 44

Generative Modelling
Given finite iid samples:

The goal is to learn to generate samples from the same distribution.

S. Diarra Mbacke (ULaval)

March 2024

6 / 44

Generative Modelling: Intuition

The goal is to learn a neural network that transforms noise into data.

S. Diarra Mbacke (ULaval)

March 2024

7 / 44

Analyzing a Generative Model

Analyzing a generative model is a challenging because:
The data-generating distribution is unknown;
Unlike supervised learning, one cannot simply compute the accuracy
on the test set;
Different ways of defining the similarity between probability measures
yield different results and have different interpretations.

S. Diarra Mbacke (ULaval)

March 2024

8 / 44

Next up
1

Context2

Preliminary Concepts
Generative models
PAC-Bayesian Theory

3

Results for GANs
Bound for Wasserstein GANs
Numerical Experiments

4

Results for VAEs
Variational Autoencoders
Bound for the Generative Model

5Conclusion

S. Diarra Mbacke (ULaval)

March 2024

9 / 44

PAC-Bayes

PAC-Bayes provides high-probability generalization bounds formachine learning models.
The theory requires very few assumptions, e.g. no assumption on thedata-generating distribution.
The bounds are numerically computable.

S. Diarra Mbacke (ULaval)

March 2024

10 / 44

PAC-Bayes: Definitions

We consider the following concepts.
An instance space X and an unknown distribution P ∗ on X .
A set S = {x1 , . . . , xn } of observations iid sampled from P ∗ .
A class H of models, called the hypothesis class.
A loss function ` : H × X → [0, ∞).
Instead of individual hypotheses h ∈ H, most PAC-Bayes bounds consider
aggregate hypotheses ρ ∈ M1+ (H).

S. Diarra Mbacke (ULaval)

March 2024

11 / 44

PAC-Bayes: Risk

Given a loss function ` : H × X → [0, ∞), the empirical and true risks ofρ ∈ M1+ (H) are defined as follows.

Empirical Risk
"R̂S (ρ) = Eh∼ρ

#n1X`(h, xi )ni=1

True Risk
R(ρ) = EE ∗ [`(h, x)]h∼ρ x∼P

S. Diarra Mbacke (ULaval)

March 2024

12 / 44

Definition: The KL divergence

Definition
Given probability distributions P, Q on H with densities p and q,
Zp(h)KL(P || Q) =dh.
p(h) logq(h)HS. Diarra Mbacke (ULaval)

March 2024

13 / 44

PAC-Bayes: Example

Theorem (Catoni (2003))
Given a distribution P ∗ over X , a hypothesis class H, a loss function` : H × X → [0, 1], a prior distribution π over H, a real number δ ∈ (0, 1),
and a real number λ > 0, with probability at least 1 − δ over the choice ofiidS ∼ P ∗ ⊗n , the following holds for any posterior distribution ρ ∈ M1+ (H):R(ρ) ≤ R̂S (ρ) +S. Diarra Mbacke (ULaval)KL(ρ || π) + log 1δλ+.
8nλMarch 2024

14 / 44

Next up
1Context

2

Preliminary Concepts
Generative models
PAC-Bayesian Theory

3

Results for GANs
Bound for Wasserstein GANs
Numerical Experiments

4

Results for VAEs
Variational Autoencoders
Bound for the Generative Model

5Conclusion

S. Diarra Mbacke (ULaval)

March 2024

15 / 44

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014)

S. Diarra Mbacke (ULaval)

March 2024

16 / 44

GANs : Definitions

We consider the following concepts.
Instance Space: X , data-generating distribution P ∗ ∈ M1+ (X ), andtraining setiidS = {x1 , . . . , xn } ∼ P ∗ .
Generator Family G: Each generator g ∈ G induces a distribution
P g ∈ M1+ (X ).
Critic Family F: A family F of functions f : X → R.

S. Diarra Mbacke (ULaval)

March 2024

17 / 44The Wasserstein DistanceDefinition
Let P, Q ∈ M1+ (X ). The Wasserstein distance between P and Q isdefined as
W1 (P, Q) = sup
E f (x) − E f (x) ,f ∈Lip1 (X ) x∼Px∼Q

where
Lip1 (X ) = {f : X → R s.t. |f (x) − f (y)| ≤ d(x, y)}.

S. Diarra Mbacke (ULaval)

March 2024

18 / 44The Wasserstein GAN

The goal is to minimize the Wasserstein distance W1 (P ∗ , P g ).
Lip1 (X ) is approximated by a subset F ⊆ Lip1 (X ) parameterized bya neural network.
The optimization objective ismin max
E ∗ [f (x)] − E g [f (x̂)] .
g ∈G f ∈Fx∼Px̂∼P

In practice, these expectations are approximated using finite samples.

S. Diarra Mbacke (ULaval)

March 2024

19 / 44

Risk
iid

iid

Given iid samples S = {x1 , . . . , xn } ∼ P ∗ and Sg = {x̂1 , . . . , x̂n } ∼ P g , letPn∗ and Png denote the corresponding empirical distributions:nPn∗ =1X
δxi
ni=1S. Diarra Mbacke (ULaval)

nandPng =1X
δx̂i .
ni=1

March 2024

20 / 44

Risk
iid

iid

Given iid samples S = {x1 , . . . , xn } ∼ P ∗ and Sg = {x̂1 , . . . , x̂n } ∼ P g , letPn∗ and Png denote the corresponding empirical distributions:nPn∗ =n1Xδxi
nandi=1

Png =1X
δx̂i .
ni=1

We define the empirical risk of a hypothesis g ∈ G as :WF (Pn∗ , P g ) = E [dF (Pn∗ , Png )]
Sg

where


dF (P, Q) = supf ∈F

S. Diarra Mbacke (ULaval)

E [f (x)] − E [f (x)] .

x∼Px∼QMarch 2024

21 / 44Theorem for Bounded Instance Spaces

The n-sized training set S is iid sampled from a distribution P ∗ on X .
Each generator g ∈ G induces a distribution P g ∈ M1+ (X ).
The prior distribution π ∈ M1+ (G) is independent of S.
λ > 0 and δ ∈ (0, 1) are some given real numbers.
The critic family F ⊆ Lip1 is symmetric.
(X , d) is a metric space with finite diameter ∆ = supx,x0 ∈X d(x, x0 ).

S. Diarra Mbacke (ULaval)

March 2024

22 / 44Theorem for WGANs

Theorem (Mbacke et al. (2023a))
The following holds with probability ≥ 1 − δ over the random draw of S,for any ρ ∈ M1+ (G):1
E E [WF (Pn∗ , P g )] ≤ E [WF (Pn∗ , P g )] +g ∼ρ Sg ∼ρλ
1λ∆2KL(ρ || π) + log+,
δ4nwhereWF (Pn∗ , P g ) = E [dF (Pn∗ , Png )] .
Sg

S. Diarra Mbacke (ULaval)

March 2024

23 / 44

Next up
1Context

2

Preliminary Concepts
Generative models
PAC-Bayesian Theory

3

Results for GANs
Bound for Wasserstein GANs
Numerical Experiments

4

Results for VAEs
Variational Autoencoders
Bound for the Generative Model

5Conclusion

S. Diarra Mbacke (ULaval)March 2024

24 / 44

We performed experiments with a WGAN on the following Gaussian
Mixtures:

Objective: Determine the order of magnitude of the numerical values ofthe bounds.

S. Diarra Mbacke (ULaval)

March 2024

25 / 44Numerical values for the ring dataset:

Numerical values for the grid dataset:

S. Diarra Mbacke (ULaval)

March 2024

26 / 44

Next up
1

Context2

Preliminary Concepts
Generative models
PAC-Bayesian Theory

3

Results for GANs
Bound for Wasserstein GANs
Numerical Experiments

4

Results for VAEs
Variational Autoencoders
Bound for the Generative Model

5Conclusion

S. Diarra Mbacke (ULaval)

March 2024

27 / 44Variational Autoencoders (VAEs) (Kingma and Welling, 2014)

S. Diarra Mbacke (ULaval)

March 2024

28 / 44Variational Autoencoders: Definitions

We consider the following concepts.
An instance space X ⊆ RD , and a data-generating distributionµ ∈ M1+ (X ).
A latent space Z = RdZ .
A prior distribution p(z) = N (0, I) on the latent space.
A posterior qφ (z|x) is parameterized by the encoder.

S. Diarra Mbacke (ULaval)

March 2024

29 / 44The Encoder

The encoder is a function"Qφ : X → R2dZ,Qφ (x) =#µφ (x),σφ (x)where the distribution qφ (z|x) = N µφ (x) , diag(σφ2 (x)) .

S. Diarra Mbacke (ULaval)

March 2024

30 / 44The Decoder

The decoder is a functiongθ : Z → X .
We assume gθ is Kθ -Lipschitz:kgθ (z1 ) − gθ (z2 )k ≤ Kθ kz1 − z2 k .

S. Diarra Mbacke (ULaval)

March 2024

31 / 44The Optimization Objective
Given a training set S = {x1 , . . . , xn }, minimize:LVAE (φ, θ) =

n

1 X


`θrec (z, xi ) +β KL(qφ (z|xi ) || p(z)).
 Enz∼qφ (z|xi ){z}|i=1 |{z}KL loss
Reconstruction loss

We define the VAE’s reconstruction loss as follows: `θrec : Z × X → [0, ∞),`θrec (z, x) = kx − gθ (z)k .

S. Diarra Mbacke (ULaval)

March 2024

32 / 44The Optimization Objective
Given a training set S = {x1 , . . . , xn }, minimize:LVAE (φ, θ) =n
1 X
`θrec (z, xi ) +β KL(qφ (z|xi ) || p(z)).
 Enz∼qφ (z|xi )|{z}i=1 |{z}KL loss
Reconstruction loss

We define the reconstruction loss as: `θrec : Z × X → [0, ∞),`θrec (z, x) = kx − gθ (z)k .

S. Diarra Mbacke (ULaval)

March 2024

33 / 44

Next up
1Context

2

Preliminary Concepts
Generative models
PAC-Bayesian Theory

3

Results for GANs
Bound for Wasserstein GANs
Numerical Experiments

4

Results for VAEs
Variational Autoencoders
Bound for the Generative Model

5Conclusion

S. Diarra Mbacke (ULaval)

March 2024

34 / 44The VAE’s generative modelOnce trained, the VAE defines the following generative model:gθ ]p(z).
Our goal is to bound the distance:
W1 (µ, gθ ]p(z)).

S. Diarra Mbacke (ULaval)

March 2024

35 / 44

Generation Guarantees for Bounded Instance Spacesµ ∈ M1+ (X ) is the data-generating distribution;iid

S = {x1 , . . . , xn } ∼ µ is a set of observed samples;p(z) = N (0, I) is the prior distribution on Z;
λ > 0 and δ ∈ (0, 1);
X has finite diameter: ∆ = supx,x0 ∈X d(x, x0 ) < ∞.

S. Diarra Mbacke (ULaval)

March 2024

36 / 44

Generation Guarantees for Bounded Instance Spaces

Theorem (Mbacke et al. (2023b))
With probability at least 1 − δ over the random draw of S ∼ µ⊗n , thefollowing holds for any posterior qφ (z|x) :n1X
W1 (µ, gθ ]p(z)) ≤ni=1

Eqφ (z|xi )1 λ 2 ∆2+ log +δ8n

S. Diarra Mbacke (ULaval)`θrec (z, xi )nKθ X+n1λ+nX

KL(qφ (z|xi ) || p(z))

i=1r2kµφ (xi )k2 + σφ (xi ) − ~1 .

i=1March 2024

37 / 44

Conclusion

Generative models are widely used in machine learning and difficult toanalyze.
PAC-Bayes is a powerful tool of statistical learning theory that can beused to analyze generative models (GANs, VAEs, diffusion models(Mbacke and Rivasplata, 2023)).
PAC-Bayes bounds for generative models are empirical, hence they
may enable new applications in practice.

S. Diarra Mbacke (ULaval)

March 2024

38 / 44ReferencesAnil, C., Lucas, J., and Grosse, R. (2019). Sorting out lipschitz function approximation.
In International Conference on Machine Learning, pages 291–301. PMLR.
Björck, Å. and Bowie, C. (1971). An iterative algorithm for computing the best estimateof an orthogonal matrix. SIAM Journal on Numerical Analysis, 8(2):358–364.
Catoni, O. (2003). A PAC-Bayesian approach to adaptive classification. preprint, LPMA
840.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S.,
Courville, A., and Bengio, Y. (2014). Generative Adversarial Nets. In Advances in
Neural Information Processing Systems, volume 27.
Kingma, D. P. and Welling, M. (2014). Auto-encoding Variational Bayes. In
International Conference on Learning Representations.
Mbacke, S. D., Clerc, F., and Germain, P. (2023a). PAC-Bayesian Generalization
Bounds for Adversarial Generative Models. In Proceedings of the 40th International
Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR.
Mbacke, S. D., Clerc, F., and Germain, P. (2023b). Statistical Guarantees for
Variational Autoencoders using PAC-Bayesian Theory. In Thirty-seventh Conference
on Neural Information Processing Systems.
Mbacke, S. D. and Rivasplata, O. (2023). A Note on the Convergence of Denoising
Diffusion Probabilistic Models. arXiv preprint arXiv:2312.05989.
S. Diarra Mbacke (ULaval)

March 2024

39 / 44

Questions?

S. Diarra Mbacke (ULaval)

March 2024

40 / 44

Experimental Details for WGANs

1
E E [WF (Pn∗ , P g )] ≤ E [WF (Pn∗ , P g )] +g ∼ρ Sg ∼ρλ
1λ∆2KL(ρ || π) + log+.
δ4nWGAN with probabilistic layers for the generator.
Lipschitz constraint with Björck Orthonormalization(Björck and Bowie,1971) and GroupSort activations (Anil et al., 2019).
We used part of the training set to learn the prior π, and theremaining part to compute the bound.
The standard deviation of the prior’s parameters
σ0 ∈ {10−7 , 10−6 , 10−5 , 10−4 , 0.001, 0.01, 0.1}.

S. Diarra Mbacke (ULaval)

March 2024

41 / 44

Reconstruction Guarantees for Bounded Instance Spacesµ ∈ M1+ (X ) is the data-generating distribution;iid

S = {x1 , . . . , xn } ∼ µ is a set of observed samples;p(z) ∈ M1+ (Z) is the prior distribution on Z;
λ > 0 and δ ∈ (0, 1);
X has finite diameter: ∆ = supx,x0 ∈X d(x, x0 ) < ∞.

S. Diarra Mbacke (ULaval)

March 2024

42 / 44

Reconstruction Guarantees for Bounded Instance Spaces

Theorem
Given a decoder θ, with probability at least 1 − δ over the random draw ofS ∼ µ⊗n , the following holds for any posterior qφ (z|x):
EEx∼µ qφ (z|x)

1`θrec (z, x) ≤nn Xi=1nEqφ (z|xi )`θrec (z, xi )+1XKL(qφ (z|xi ) || p(z))
λi=11
λ∆2
1.
+ log + Kφ Kθ ∆ +λδ
8n

S. Diarra Mbacke (ULaval)

March 2024

43 / 44

Regenerated Distribution

Defineµ̂φ,θ = n1Pn

i=1 gθ ]qφ (z|xi ).

The triangle inequality implies
W1 (µ, gθ ]p(z)) ≤ W1 (µ, µ̂φ,θ ) + W1 (µ̂φ,θ , gθ ]p(z)).

S. Diarra Mbacke (ULaval)

March 2024

44 / 44