Institutdes algorithmes
d’apprentissagede Montréal

Deep Generative Models
Aaron Courville
MILA, Université de Montréal

6.S191: Introduction to Deep Learning
MIT, Jan 30th,2018
1•Density estimation

Generative modeling
Generative Modeling•

GenerativeModeling
Generative models take training samples from some data
distribution andlearn a estimation
model that represents that distribution.
• Density••

Samplegeneration
Figure1: Some generative models perform density estimation. These models take a

Generative Modelingtraining set of examples drawn from an unknown data-generating distribution pdata
and return an estimate of that distribution. The estimate pmodel can be evaluated fora particular value of x to obtain an estimate pmodel (x) of the true density pmodel (x).
This figure illustrates the process for a collection of samples of one-dimensional data
and a Gaussian model.

Density estimation:••

Density estimation••• Density
Sample estimation
generation

Samplegeneration
Figure1: Some generative models perform density estimation. These models take a

Sample generation:•training set of examples drawn from an unknown data-generating distribution pdata
and return an estimate of that distribution. The estimate pmodel can be evaluated fora particular value of x to obtain an estimate pmodel (x) of the true density pmodel (x).
This figure illustrates the process for a collection of samples of one-dimensional data
and a Gaussian model.

Sample generation

Training examples

Training examples

Model samples

Model samples(Goodfellow(Goodfellow 2016)

imagessamples
taken from
Figure 2: Some generative models are able to generate
fromGoodfellow
the model (2017)
distri- 2

Why generative models?
• Many tasks require structured output-Eg. Machine translation

image credit: Adam Geitgey blog (2016) Machine Learning is Fun Part 5: Language Translation with Deep Learning and the Magic of Sequences

Why Generative Models? Outlier detectioncars• Large-scale deployment of CNN-

wheelchairs

based perception systems isbecoming a reality.
• How do we detect when weencounter something new or rare(i.e. not appearing in the trainingdata)?
• Goal: detect these outliers(anomalies) to avoid dangerous
misclassification.
• Strategy: Leverage generativemodels of the training distribution
to detect outliers.

✘✘Outlier!

✘Why Generative Models? Generation for Simulation•

Supports Reinforcement Learning for Robotics: Make simulations sufficiently
realistic that learned policies can readily transfer to real-world application

Generative model

Photo from IEEE Spectrum

5

Deep Generative Models: Outline
Autoregressive models•

Deep NADE, PixelRNN, PixelCNN, WaveNet, Video Pixel
Network, etc.

Latent variable models•Variational Auto encoders•

Generative Adversarial Networks
our focus today6

Latent Variable Models• The Variational Autoencoder model:-Kingma and Welling, Auto-Encoding Variational Bayes, International Conference on Learning
Representations (ICLR) 2014.

-Rezende, Mohamed and Wierstra, Stochastic back-propagation and variational inference in deeplatent Gaussian models. ICML 2014.

x2z2gz1x3x1

Image from: Ward, A. D., Hamarneh, G.: 3D Surface Parameterization Using Manifold Learning for Medial Shape Representation, Conference on Image Processing, Proc. of SPIE Medical Imaging, 20077xz

Latent Variable Models
z222gFrey Faces:

x1x1ExpressionzMNIST:z3(a) Learned Frey Face manifoldz
1Pose(a) Learned Frey Face manifold(b) Learned MNIST manifold(b) Learned MNIST manifoldz8

Latent Variable Models• latent variable model: learn a mapping from some latent variable z to a complicateddistribution on x.
!

p(x) =p(x, z) dz

where p(x, z) = p(x | z)p(z)p(z) = something simple

p(x | z) = g(z)• Can we learn to decouple the true explanatory factors underlying the data distribution?
E.g. separate identity and expression in face images

x2

z2gz1

x3

x1

Image from: Ward, A. D., Hamarneh, G.: 3D Surface Parameterization Using Manifold Learning for Medial Shape Representation, Conference on Image Processing, Proc. of SPIE Medical Imaging, 20079

Latent Variable Models• latent variable model: learn a mapping from some latent variable z to a complicateddistribution on x.
!

p(x) =p(x, z) dz

where p(x, z) = p(x | z)p(z)p(z) = something simple

p(x | z) = g(z)• Can we learn to decouple the true explanatory factors underlying the data distribution?
E.g. separate identity and expression in face images
x2z2z:g(z):gz1x3x1x:

Image from: Ward, A. D., Hamarneh, G.: 3D Surface Parameterization Using Manifold Learning for Medial Shape Representation, Conference on Image Processing, Proc. of SPIE Medical Imaging, 2007 10

Variational Auto-Encoder (VAE)•

Where does z come from? — The classic DAG problem.

•The VAE approach: introduce an inference machine qφ (z | x) thatlearns to approximate the posterior pθ (z | x).

•Define a variational lower bound on the data likelihood: pθ (x) ≥ L(θ, φ, x)L( , , x) = Eq (z|x) [log p (x, z)

log q (z | x)]= Eq (z|x) [log p (x | z) + log p (z)=•What is qφ (z | x) ?

log q (z | x)]

DKL (q (z | x) p (z)) + Eq (z|x) [log p (x | z)]
regularization termreconstruction term
11

VAE Inference model•

The VAE approach: introduce an inference model qφ (z | x) that learns toapproximates the intractable posterior pθ (z | x) by optimizing the variational
lower bound:L(θ, φ, x) = −DKL (qφ (z | x)∥ pθ (z)) + Eqφ (z|x) [log pθ (x | z)]•We parameterize qφ (z | x) with another neural network:qφ (z | x) = q(z; f (x, φ))pθ (x | z) = p(x; g(z, θ))f(x):g(z):

z:

x:

z:

x:
12

Reparametrization trick•

Adding a few details + one really important trick•

Let’s consider z to be real and qφ (z | x) = N (z; µz (x), σz (x))•Parametrize z as z = µ (x) + σ (x)ϵz

Encoder:f(z):x:zz = N (0, 1)

σz (x){{µz (x)z

where ϵDecoder: z :g(z):x:13

Training with backpropagation!
•Due to a reparametrization trick, we can simultaneously train both the generativemodel pθ (x | z) and the inference model qφ (z | x) by optimizing the variationalbound using gradient backpropagation.

Objective function: L(θ, φ, x) = −DKL (qφ (z | x)∥ pθ (z)) + Eqφ (z|x) [log pθ (x | z)]
Forward propagationzx

Backward propagationqφ (z | x)pθ (x | z)x̂14

vanilla VAE samples

Labelled Faces in the Wild (LFW)

Impressive …
… at the timeImageNet (small)

15

PixelVAE•

Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed Adrien Ali Taiga,
Francesco Visin, David Vazquez, Aaron Courville. ICLR 2017

Uses a PixelCNN in the VAE decoder to help avoid the blurring
caused by the standard VAE assumption of independent pixels.

PixelCNN
PixelCNN
PixelCNN

PixelVAE Samples

LSUN bedroom scenes (64x64)(Gulrajani et al. 2017)

ImageNet (64x64)
17Inverse Autoregressive Flow (Kingma et al., NIPS 2016)(a) Prior distribution••(b) Posteriors in standard VAE (c) Posteriors in VAE with IAF

Figure 1: Best viewed in color. We fitted a variational auto-encoder (VAE) with a spherical Gaussianprior, and VAE
with factorized
Gaussianposteriors (b)- orinverse autoregressive
flowcan(IAF)
posteriors (c) fit
Standardposteriors
are factorized
limiting
how well they(marginally)toa
toy
datasetwithfourdatapoints.
Eachcoloredclustercorresponds
totheposterior
distributionofthe prior.
one datapoint. IAF greatly improves the flexibility of the posterior distributions, and allows for amuch better fit between the posteriors and the prior.

IAF greatly improves the flexibility of the posterior distributions, and allows for amuch better fit between the posteriors and the prior.
improving inference models including previously used normalizing flows, this transformation is well
suited to high-dimensional tensor variables, such as spatio-temporally organized variables.

18

Another way to train a latent variable model?
inference
Latent variables

zz2x2

Gz?

Gx3xz1

x1

Observed variables

19xGenerative Adversarial Networks
D tries tooutput 1

D tries tooutput 0

Differentiable
function D

Differentiablefunction D

x sampledfrom datax sampledfrom model

Differentiable
function GInput noiseZ20

Generative Adversarial NetworksD(x)DataDiscriminatorG(z)x ∼ PrGeneratorz ∼ p(z)x̃ ∼ Pg

21

GAN Objective•

Formally, express the game between discriminator D andgenerator G with the minimax objective:

min max E [log(D(x))] + E [log(1 − D(x̃))].
GDx∼Prx̃∼Pg

where:- Pr is the data distribution- Pg is the model distribution implicitly defined by:x̃ = G(z),-z ∼ p(z)the generator input z is sampled from some simple
noise distribution, (e.g. uniform or Gaussian).

22

GAN Theory•

Optimal (nonparametric) discriminator:p(x)r∗
D (x) =pr (x) + pg (x)•Under an ideal discriminator, the generator minimizes theJensen-Shannon divergence between Pr and Pg.
! "#! "#" Pr + Pg
" Pr + Pg""
JS(Pr ∥Pg ) = KL Pr "+ KL Pg "22"#!
pr (x)pr (x)dµ(x)where KL(Pr ∥Pg ) = logpg (x)23

GAN Theory … in practice•The minimax objective leads to vanishing gradients as thediscriminator saturates.

•In practice, Goodfellow et al (2014) advocate the heuristic
training objective:
max E [log(D(x))] + E [log(1 − D(x̃))].
Dx∼Prx̃∼Pg

max E [log(D(x̃))].
G‣x̃∼PgHowever, this modified loss function can still misbehave inthe presence of a good discriminator.
24

GAN samples

MNIST

CIFAR-1025

Least-Squares GAN

Xudong Mao, Qing Li†, Haoran Xie, Raymond
Y.K. Lau and Zhen Wang, ArXiv, Feb. 2017

128x128
LSUNbedroom
scenes(a) Generated by LSGANs.
26

Under review as a conference paper at ICLR 2016

DCGAN samples (Radford, Metz and Chintala; 2016)
Z-space interpolations

LSUN bedroom scenes

27

What makes GANs special?
x2

Cartoon of the Image manifold:x1

28

What makes GANs special?
x2

x2

x1
more traditional max-likelihood approach

x1

GAN

29

But what about inference…
Can we incorporate an inference mechanism into GANs?

•Latent variables

zz2

x2

Gz?

Gx3
xz1

x1

Observed variables

30

x

ALI / BiGAN: model diagram
Prior distributionz ~ p(z)D(x, z)x ~ q(x)Gx(z)
DecoderGz(x)Encoderz ~ q(z | x)x ~ p(x | z)

Data distribution• ALI: Vincent Dumoulin, Ishmael Belghazi, Olivier Mastropietro, Ben Poole, Alex Lamb, Martin Arjovsky (2016)

ADVERSARIALLY LEARNED INFERENCE, arXiv:1606.00704, ICLR 2017
• BiGAN: Donahue, Krähenbühl and Darrell (2016), ADVERSARIAL FEATURE LEARNING, arXiv:1605.09782, ICLR 201731

Hierarchical ALI
CelebA-128X128

Model samples

32

ReconReconstructions given z1, z2

DataRecon

Data

Hierarchical ALI: CelebA-128X128

Reconstructions given z233

cycleGAN: Adversarial training of domain transformations(Zhu et al. ICCV 2017)•CycleGAN learns transformations across domains with unpaired data.

•Combines GAN loss with “cycle-consistency loss”: L1 reconstruction.

Image credits: Jun-Yan Zhu*, Taesung Park*, Phillip Isola, and Alexei A. Efros. "Unpaired Image-to-Image Translation using
Cycle-Consistent Adversarial Networks", in IEEE International Conference on Computer Vision (ICCV), 2017.

CycleGAN for unpaired data

Image credits: Jun-Yan Zhu*, Taesung Park*, Phillip Isola, and Alexei A. Efros. "Unpaired Image-to-Image Translation using
Cycle-Consistent Adversarial Networks", in IEEE International Conference on Computer Vision (ICCV), 2017.

Under review as a conference paper at ICLR 2018

PROGRESSIVE GROWING OF GANS FOR IMPROVEDQUALITY, STABILITY, AND VARIATION (Kerras et al. from NVIDIA, 2017)•

Recent work fromNVIDIA.

•Improves image quality
by growing the model
size throughout training.

•Samples from a modeltrained on the CelebA
face dataset.

1024x1024
modelsamples
Figure 5: 1024 ⇥ 1024 images generatedusing the C
ELEBA-HQdataset

Under review as a conference paper at ICLR 2018

PROGRESSIVE GROWING OF GANS FOR IMPROVEDQUALITY, STABILITY, AND VARIATION (Kerras et al. from NVIDIA, 2017)

GLatent

LatentLatent4x44x48x84x4

RealsReals

D…

1024x1024Reals
1024x1024

4x48x84x44x4
Training progresses

Under review as a conference paper at ICLR 2018

PROGRESSIVE GROWING OF GANS FOR IMPROVEDQUALITY, STABILITY, AND VARIATION (Kerras et al. from NVIDIA, 2017)•

Recent work fromNVIDIA.

•Improves image quality
by growing the modelsize throughout training.

•Conditional samples
from a model trained on
the LSUN dataset

POTTEDPLANT

HORSE

SOFA

BUS

CHURCHOUTDOORBICYCLETVMONITORThank You!
Questions?
Model samples

39