Deep Generative Models
MIT 6.S191
Alexander Amini
January 29, 2019

Which face is fake?

6.S191 Introduction to Deep Learning
introtodeeplearning.com[1]

1/29/19Supervised vs unsupervised learning
Supervised Learning

Unsupervised Learning

Data: (", $)" is data, $ is label

Data: "" is data, no labels!

Goal: Learn function to map"→$Goal: Learn some hidden or
underlying structure of the data

Examples: Classification,regression, object detection,
semantic segmentation, etc.

Examples: Clustering, feature ordimensionality reduction, etc.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19Supervised vs unsupervised learning
Supervised Learning

Unsupervised Learning

Data: (", $)" is data, $ is label

Data: "" is data, no labels!

Goal: Learn function to map"→$

Goal: Learn some hidden or
underlying structure of the data

Examples: Classification,regression, object detection,semantic segmentation, etc.

Examples: Clustering, feature ordimensionality reduction, etc.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Generative modeling
Goal: Take as input training samples from some distribution
and learn a model that represents that distribution
Sample Generation

Density Estimation

Input samples

Generated samples

Training data ~ "#$%$ &Generated ~ "'(#)* &samples

How can we learn "'(#)* & similar to "#$%$ & ?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Why generative models? Debiasing
Capable of uncovering underlying latent variables in a dataset

vsHomogeneous skin color, pose

Diverse skin color, pose, illumination

How can we use latent distributions to create fair and representative datasets?
6.S191 Introduction to Deep Learning
introtodeeplearning.com[2]

1/29/19

Why generative models? Outlier detection• Problem: How can we detect when
we encounter something new or rare?
• Strategy: Leverage generativemodels, detect outliers in thedistribution• Use outliers during training toimprove even more!

95% of Driving Data:(1) sunny, (2) highway, (3) straight road

Detect outliers to avoid unpredictable behavior when training

Edge Cases
6.S191 Introduction to Deep Learning
introtodeeplearning.com

Harsh WeatherPedestrians[3]

1/29/19

Latent variable modelsAutoencoders and Variational
Autoencoders (VAEs)

Generative Adversarial
Networks (GANs)

What is a latent variable?

Myth of the Cave
6.S191 Introduction to Deep Learning
introtodeeplearning.com[4]

1/29/19

What is a latent variable?

Can we learn the true explanatory factors, e.g. latent variables, from only observed data?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Autoencoders

Autoencoders: background
Unsupervised approach for learning a lower-dimensional feature
representation from unlabeled training data
Why do we care about alow-dimensional "?

!

"“Encoder” learns mapping from the data, !, to a low-dimensional latent space, "6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Autoencoders: backgroundHow can we learn this latent space?
Train the model to use these features to reconstruct the original data!

"!#“Decoder” learns mapping back from latent, ", to a reconstructed observation, !#6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Autoencoders: backgroundHow can we learn this latent space?
Train the model to use these features to reconstruct the original data!

"!#ℒ !, !# = ! − !# (Loss function doesn’t
use any labels!!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Autoencoders: backgroundHow can we learn this latent space?
Train the model to use these features to reconstruct the original data!

"!#ℒ !, !# = ! − !# (Loss function doesn’t
use any labels!!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Dimensionality of latent space àreconstruction quality
Autoencoding is a form of compression!
Smaller latent space will force a larger training bottleneck
2D latent space

5D latent space

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Ground Truth[5]

1/29/19

Autoencoders for representation learning
Bottleneck hidden layer forces network to learn a compressed
latent representation
Reconstruction loss forces the latent representation to capture(or encode) as much “information” about the data as possible
Autoencoding = Automatically encoding data
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Variational Autoencoders (VAEs)

VAEs: key difference with traditional autoencoder!

"

6.S191 Introduction to Deep Learning
introtodeeplearning.com!#

1/29/19

VAEs: key difference with traditional autoencoder

$!

"!#%

6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

VAEs: key difference with traditional autoencoder
meanvector

$!

"!#%
standard deviation
vectorVariational autoencoders are a probabilistic twist on autoencoders!
Sample from the mean and standard dev. to compute latent sample
6.S191 Introduction to Deep Learningintrotodeeplearning.com[6]

1/29/19

VAE optimization

$!

"!#%

Encoder computes: &' (z|!)

Decoder computes: ,- (x|")

6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

VAE optimization

$!

"!#%

Encoder computes: &' (z|!)

Decoder computes:,- (x|")ℒ ϕ, 2 = (reconstruction loss) + (regularization term)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

VAE optimization

$!

"!#%

Encoder computes: &' (z|!)

Decoder computes:,- (x|")ℒ ϕ, 2, ! = (reconstruction loss) + (regularization term)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

VAE optimization
e.g. ! − !# 5$!

"!#%

Encoder computes: &' (z|!)

Decoder computes:,- (x|")ℒ ϕ, 2, ! = (reconstruction loss) + (regularization term)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

VAE optimization
Inferred latent
distribution

Fixed prior on
latent distribution

4 &' z ! ∥ & "

$!

"!#%

Encoder computes: &' (z|!)

Decoder computes:,- (x|")ℒ ϕ, 2, ! = (reconstruction loss) + (regularization term)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

Priors on the latent distribution! "# z % ∥ " 'Inferred latent
distribution

Fixed prior on
latent distribution

Common choice of prior:" ' = ) * = 0, - . = 1• Encourages encodings to distribute encodings evenly aroundthe center of the latent space• Penalize the network when it tries to “cheat” by clustering
points in specific regions (ie. memorizing the data)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[7]

1/29/19

Priors on the latent distribution! "# z % ∥ " '678

1= − 2 -3 + *3. − 1 − log -3
2KL-divergence between
the two distributions

345

Common choice of prior:" ' = ) * = 0, - . = 1• Encourages encodings to distribute encodings evenly aroundthe center of the latent space• Penalize the network when it tries to “cheat” by clustering
points in specific regions (ie. memorizing the data)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[7]

1/29/19

VAEs computation graph$!

"!#%Encoder computes: &' (z|!)

Decoder computes:,- (x|")ℒ ϕ, 2, ! = (reconstruction loss) + (regularization term)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

VAEs computation graph
Problem: We cannot backpropagate gradients through sampling layers!

$!

"!#%

Encoder computes: &' (z|!)

Decoder computes:,- (x|")ℒ ϕ, 2, ! = (reconstruction loss) + (regularization term)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

Reparametrizing the sampling layer
Key Idea:! ~%(", # ()Consider the sampled latentvector as a sum of• a fixed " vector,• and fixed # vector, scaled byrandom constants drawn fromthe prior distribution"!
#⇒ ! = " + #⨀.
where .~%(0,1)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

Reparametrizing the sampling layer!

Backprop

Deterministic node

Stochastic node

" ∼ $% (z|))"+)

Original form
6.S191 Introduction to Deep Learning
introtodeeplearning.com[6]

1/29/19

Reparametrizing the sampling layer
Backprop!

0!
0"Deterministic node

" ∼ $% (z|))

Stochastic node"!

"" = ,(-, ), /)

)

/0!
0--)

-

Original form
6.S191 Introduction to Deep Learning
introtodeeplearning.com~2(0,1)

Reparametrized form
[6]

1/29/19

VAEs: Latent perturbation
Slowly increase or decrease a single latent variable
Keep all other variables fixed

Head pose
Different dimensions of ! encodes different interpretable latent features
6.S191 Introduction to Deep Learning
introtodeeplearning.com[8]

1/29/19

VAEs: Latent perturbation
Ideally, we want latent variables thatare uncorrelated with each other
Enforce diagonal prior on the latentvariables to encourage
independence

SmileDisentanglement
Head pose
6.S191 Introduction to Deep Learning
introtodeeplearning.com[8]

1/29/19

VAEs: Latent perturbation

Google BeatBlender
6.S191 Introduction to Deep Learning
introtodeeplearning.com[9]

1/29/19

VAEs: Latent perturbation

6.S191 Introduction to Deep Learning
introtodeeplearning.com[10] 1/29/19

VAE summary
1. Compress representation of world to something we can use to learn!

"

6.S191 Introduction to Deep Learning
introtodeeplearning.com!#

1/29/19

VAE summary
1. Compress representation of world to something we can use to learn
2. Reconstruction allows for unsupervised learning (no labels!)!

"

6.S191 Introduction to Deep Learning
introtodeeplearning.com!#

1/29/19

VAE summary
1. Compress representation of world to something we can use to learn
2. Reconstruction allows for unsupervised learning (no labels!)
3. Reparameterization trick to train end-to-end!

"

6.S191 Introduction to Deep Learning
introtodeeplearning.com!#

1/29/19

VAE summary
1.
2.
3.
4.

Compress representation of world to something we can use to learn
Reconstruction allows for unsupervised learning (no labels!)
Reparameterization trick to train end-to-end
Interpret hidden latent variables using perturbation!

"

6.S191 Introduction to Deep Learning
introtodeeplearning.com!#

1/29/19

VAE summary
1.
2.
3.
4.
5.

Compress representation of world to something we can use to learn
Reconstruction allows for unsupervised learning (no labels!)
Reparameterization trick to train end-to-end
Interpret hidden latent variables using perturbation
Generating new examples!

"

6.S191 Introduction to Deep Learning
introtodeeplearning.com!#1/29/19

Generative Adversarial Networks (GANs)

What if we just want to sample?
Idea: don’t explicitly model density, and instead just sample to generate new instances.
Problem: want to sample from complex distribution – can’t do this directly!

noise!

"#$%&'

Solution: sample from something simple (noise), learn atransformation to the training distribution.

“fake” sample from thetraining distribution

Generator Network "6.S191 Introduction to Deep Learningintrotodeeplearning.com[11] 1/29/19

Generative Adversarial Networks (GANs)$%&'(Generative Adversarial Networks (GANs) are a way to make a generativemodel by having two neural networks compete with each other.

#!

"+$)'*&The generator turns noise into an imitationof the data to try to trick the discriminator.

noise

The discriminator tries to identify real
data from fakes created by the generator.

6.S191 Introduction to Deep Learning
introtodeeplearning.com[11] 1/29/19

Intuition behind GANs
Generator starts from noise to try to create an imitation of the data.

Generator

Fake data
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator looks at both real data and fake data created by the generator.

Discriminator

Generator

Fake data
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator looks at both real data and fake data created by the generator.

Discriminator

Generator

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator tries to predict what’s real and what’s fake.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator tries to predict what’s real and what’s fake.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator tries to predict what’s real and what’s fake.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learningintrotodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator tries to predict what’s real and what’s fake.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Generator tries to improve its imitation of the data.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Generator tries to improve its imitation of the data.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Generator tries to improve its imitation of the data.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator tries to predict what’s real and what’s fake.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator tries to predict what’s real and what’s fake.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator tries to predict what’s real and what’s fake.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator tries to predict what’s real and what’s fake.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Generator tries to improve its imitation of the data.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Generator tries to improve its imitation of the data.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Generator tries to improve its imitation of the data.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Intuition behind GANs
Discriminator tries to identify real data from fakes created by the generator.
Generator tries to create imitations of data to trick the discriminator.

Discriminator

Generator! "#$% = 1

Real data

Fake data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Training GANs
Discriminator tries to identify real data from fakes created by the generator.
Generator tries to create imitations of data to trick the discriminator.

Train GAN jointly via minimax game:min max )*~,(-.- log 2$( 3 + )5~,(5) log 1 − 2$( :$% (;)$%$(Discriminator wants to maximize objective s.t. 2 3 close to 1, 2 :(; ) close to 0.
Generator wants to minimize objective s.t. 2 :(; ) close to 1.

6.S191 Introduction to Deep Learning
introtodeeplearning.com[11] 1/29/19

Why GANs?

A. Courville, 6S191 2018.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Why GANs?

A. Courville, 6S191 2018.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Generating new data with GANs$%&'(After training, use generator network to create new data that’s never been seen before.

noise!

"+$)'*&#

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

GANs: Recent Advances

Progressive growing of GANs (NVIDIA)

Karras et al., ICLR 2018.
6.S191 Introduction to Deep Learning
introtodeeplearning.com[12] 1/29/19

Progressive growing of GANs: results

Karras et al., ICLR 2018.
6.S191 Introduction to Deep Learning
introtodeeplearning.com[12] 1/29/19

Style-based generator: results

Karras et al., Arxiv 2018.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Style-based transfer: results

Karras et al., Arxiv 2018.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

CycleGAN: domain transformation
CycleGAN learns transformations across domains with unpaired data.

Zhu et al., ICCV 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Deep Generative Modeling: Summary
Autoencoders and Variational
Autoencoders (VAEs)

Generative Adversarial
Networks (GANs)

Learn lower-dimensional latent
space and sample to generate
input reconstructions

Competing generator anddiscriminator networks

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19References:https://goo.gl/ZuBkGx9