Generative Adversarial Networks
Petar Veličković
Artificial Intelligence Group
Department of Computer Science and Technology, University of Cambridge, UK
Pint of Science: I think, therefore AI

15 May 2018

Introduction

I In this talk, I will guide you through a high level overview ofGenerative Adversarial Networks (GANs)—one of the mostpopular ideas to hit deep learning in the past decade.
I This will involve a journey through the essentials of how neuralnetworks normally work. . .
I . . . followed by a dive into turning horses into zebras,

generating fake celebrities, and anecdotes about howpints can lead to wonderful science.

Motivation: notMNISTI Which characters do you see? (How did you conclude this?)

I Imagine someone asked you to write a program thatrecognises characters from arbitrary glyphs. . .

Intelligent systems

I Although the previous task was likely simple to you, you(probably) couldn’t turn your thought process into a concise
sequence of instructions for a program!
I Unlike a “dumb” program (that just blindly executes

preprogrammed instructions), you’ve been exposed to a lot of A
characters during your lifetimes, and eventually “learnt” thecomplex features making something an A!
I Desire to design such systems (capable of generalising frompast experiences) is the essence of machine learning!
I

How many such systems do we know from nature?

Specialisation in the brain
I We know that different parts of the brain perform different tasks:

I There is increasing evidence that the brain:
II

Learns from exposure to data;
Is not preprogrammed!

Brain & data

I The majority of what we know about the brain comes fromstudying brain damage:Rerouting visual inputs into the auditory region of baby ferrets
makes this region capable of dealing with visual input!
I As far as we know (for now), the modified region works equally
good as the visual cortex of healthy ferrets!
II If there are no major biological differences in learning toprocess different kinds of input. . .
I=⇒ the brain likely uses a general learning algorithm, capableof adapting to a wide spectrum of inputs.

I We’d very much like to capture this algorithm!

A real neuron!

An artificial neuron!
Within this context sometimes also called a perceptron (. . . )

x2
x3..
.
xn

w2

b+ΣnPwi xi

i=1w3
n

x1

bw1w

+1

We train the neuron by exposing it to many (~x , y) examples, and~ and b to make its output close to y when given ~xthen adjusting w
as input.

Neural networks and deep learning

I It is easy to extend a single neuron to a neural network—simply

connect outputs of neurons to inputs of other neurons.
I Typically we organise neural networks in a sequence of layers,

such that a single layer only processes output from theprevious layer.
I By extension from a single neuron, we train the neural networkby exposing it to many input-output pairs, and appropriately
adjusting the parameters of all the neurons.

Multilayer perceptrons
The most potent feedforward architecture allows for full connectivity
between layers—sometimes also called a multilayer perceptron.
Input
layer
+1

bx1

w1

w3
nx3..
.
xn

w2

w

x2


nP
σ b+wi xi
Σσi=1

I1

Hidden Outputlayerlayer
O1

I2
I3O2

Neural network depthI I’d like to highlight a specific parameter: the number of hidden

layers, i.e. the network’s depth.
I What do you think, how many hidden layers are sufficient tolearn any “useful” function?

Neural network depth

I I’d like to highlight a specific parameter: the number of hidden

layers, i.e. the network’s depth.
I What do you think, how many hidden layers are sufficient tolearn any “useful” function?
I One! (Cybenko’s theorem, 1989.)

Neural network depth

I I’d like to highlight a specific parameter: the number of hiddenlayers, i.e. the network’s depth.
I What do you think, how many hidden layers are sufficient tolearn any “useful” function?
I One! (Cybenko’s theorem, 1989.)
I However, the number of neurons in this layer would need to be

astronomical for almost all problems we care about.
I We must go deeper. . .
IIEvery network with > 1 hidden layer is considered deep!
Today’s state-of-the-art networks often have over 150 layers.

Deep neural networks
O1

O2

×∞

I1

I2

I3

Quiz: What do we have here?

DeepBlue vs. AlphaGo
I Main idea (roughly) the same: assume that a grandmaster isonly capable of thinking k steps ahead—then generate a(near-)optimal move when considering k 0 > k steps ahead.
IDeepBlue does this exhaustively, AlphaGo sparsely (discarding
many “highly unlikely” moves).

I One of the key issues: when stopping exploration, how do wedetermine the advantage that player 1 has?
DeepBlue: Gather a team of chess experts, and define a function
f : Board → R, to define this advantage.
AlphaGo: Feed the raw state of the board to a deep neural network, andhave it learn the advantage function by itself.
I This highlights an important paradigm shift brought about by

deep learning. . .

Feature engineering

I Historically, machine learning problems were tackled bydefining a set of features to be manually extracted from raw
data, and given as inputs for “shallow” models.
Many scientists built entire PhDs focusing on features of interestfor just one such problem!
I Generalisability: very small (often zero)!
II With deep learning, the network learns the best features byitself, directly from raw data!
For the first time connected researchers from fully distinct areas,e.g. natural language processing and computer vision.
I =⇒ a person capable of working with deep neural networks
may readily apply their knowledge to create state-of-the-art
models in virtually any domain (assuming a large dataset)!
IRepresentation learning

I As inputs propagate through the layers, the network captures

more complex representations of them.
I It will be extremely valuable for us to be able to reason about

these representations!
I Typically, models that deal with images will tend to have thebest visualisations.

Passing data through the network: Input
O1

I1

O2

I2

I3

Passing data through the network: Shallow layer
O1

I1

O2

I2

I3

Passing data through the network: Deep layer
O1

I1

O2

I2

I3

Passing data through the network: Output
O1

I1

O2

I2

I3

We’re good at detecting what’s in an image. . .

. . . even exceeding human performance. . .

. . . or are we???

The importance of understanding data

Generative modelling“What I cannot create, I do
not understand.”—Richard Feynman

Generative Adversarial Networks

I “The most important one, in my opinion, is adversarial training(also called GAN for Generative Adversarial Networks). This,and the variations that are now being proposed is the mostinteresting idea in the last 10 years in ML, in my opinion.”—Yann LeCun(Director of Research, Facebook AI Research)

GANs are everywhere

The generator

I Imagine that we had a neural network, G, capable ofgenerating new data from random inputs ~z .. . .
noiseG(~z )~zgenerator~xWe will call this neural network the generator.
I What we want to do is make the output of this generator match

the properties of real data.
I But how can we check if a synthetic input has properties similar

to real data?

The discriminator

I Checking for properties by looking over full data (e.g. individual

image pixels) is hard.
I We can utilise a neural network to extract underlying featuresfrom inputs, and examine those.
I If these features are “good enough”, the network should becapable of telling real and synthetic inputs apart!
I Call this network the discriminator, D(~x ).
I Essentially, a binary classifier, telling whether ~x is real orsynthetic. N.B. The discriminator effectively specifies theerror that we want the generator to minimise!

The GAN framework (Goodfellow et al., 2014)

real data~xrealD(~x )~xnoiseG(~z )~zgenerator

discriminatorreal?

~xfakeTwo neural networks playing a game. . .
Alternate updating their weights; hopefully they improve together!

The GAN framework—update step 1

real data~xrealD(~x )~xnoiseG(~z )~zgenerator

discriminator~xfakeTrain discriminator to say ‘real’ on real data.

real?

The GAN framework—update step 2

real data~xrealD(~x )~xnoiseG(~z )~zgenerator

discriminator~xfakeTrain discriminator to say ‘fake’ on fake data.

real?

The GAN framework—update step 3

real data~xrealD(~x )~xnoiseG(~z )~zgenerator

discriminatorreal?

~xfakeTrain generator to make discriminator say ‘real’ on fake data.

The desired final outcome

min max V (D, G) = E~x ∼pdata (~x ) [log D(~x )] + E~z ∼pθ (~z ) [log(1 − D(G(~z )))]GDI Assuming everything goes well, concluding the training

process we obtain two very useful networks!
The generator, G(~z ), becomes capable of generating extremely
useful synthetic examples.
I The discriminator, D(~x ), becomes a high-quality feature
extractor from data.
II Let’s see how good these synthetic examples from G(~z)

actually get. . .

Generating fake celebrities (Karras et al., 2018)

A step further—domain transfer

I Similar ideas can be applied in order to transform data fromone type to another. . .
I We learn two generators, GAB and GBA , to transfer inputs

from type A to type B, and vice-versa.
I We also learn the two discriminators, DA and DB .
I Besides the usual GAN game, we also can enforce cycle

consistency: GBA (GAB (~a)) ∼ ~a.
I

This means that if we transform an input of type A to type B,
then transform the result back into type A, we should be able torecover the original input.

We arrive at the remarkably powerful CycleGAN model. . .

Domain transfer: CycleGAN (Zhu et al., 2017)

real data(type B)~breal~b

real data(type A)

~arealGAB (~a)generator(A → B)~bfakeGBA (~b)generator(B → A)DB (~b)discriminator(type B)~arecreal?

Domain transfer: CycleGAN (Zhu et al., 2017)

Where did this all begin?

A true pint of science. . .

. . . credited in the paper.

An overview of historical deep learning ideas

I Initially, we needed to extract hand-crafted features before

applying a machine learning model to them.
IDeep neural networks can perform feature extraction bythemselves.

I Then, we needed to decide on how to estimate the error of ournetwork’s prediction.
IGANs use a neural network (the discriminator) to compute this!

I We need to figure out a correct way to reduce this error.
ILearn how to learn?

Thank you!

Questions?
petar.velickovic@cst.cam.ac.uk