Deep Learning
Nando de Freitas

O X

Place cells in the hippocampus and grid cells in the entorhinal cortex[John O’Keefe, May-Britt Moser and Edvard Mosel, NOBEL 2014]

Distributed representations have exponential power

Feature vector

10

1

00

Hidden units

Learnedparameters
form images
4x4 imageDistributed representations enable us to obtain
continuous embeddings in metric spaces

Feature vector(embedding)

1.1

0.8

7.1

0.2 … 0.0

Hidden units
Learned parameters
form images

Eco-statistics

Neocognitron (Fukushima)

Convolutional networks

Google’s neural net learns just by watching youtube videos

[Matthew Zeiler & Rob Fergus]

Predicting parameters: Intuition

Reconstruction ICA
STL-10MLP
MNISTConvolutional netCIFAR-10

Fast semantic hashing / embedded devices

Multi-layer perceptron for speech[Denil, Shakibi, Dinh, Ranzato & NdF, NIPS 2013]

Low-rank fast convolutions[Jaderberg, Vedaldi & Zisserman, 2014][Denton, Zaremba, Bruna, LeCun & Fergus, 2014]

Fast-food convnets

We reduce the original 405,000
parameters to a mere 40,960

Machines that learn to recognise what they see and hear are at theheart of Apple, Google, Amazon, Facebook, Netflix, Microsoft, etc.

Structured queries and outputs
Who’s likely
to want
to watch thismovie withme onFriday?
ooooooooooooooooooooooooo
Phil is available and he likes movies with Downey JR

ConvNets and Recursive Nets

[Cho et al, 2014][Kalchbrenner et al, 2014][Richard Socher et al, …]Sentence ConvNets[Kalchbrenner, Grefenstette, Blunsom, 2014]

Sentence
DynConvNet[Kalchbrenner et al, 2014]

Sentence models[Kalchbrenner, Grefenstette, Blunsom, 2014]

Applications: Translation[Kalchbrenner, Blunsom, 2013]

Transfer: Question answering[Grefenstette, Blunsom, NdF, Hermann, 2014]

Document models[Denil, Demiraj, Kalchbrenner, Blunsom & NdF, 2014]

Document model: summarisation
Saliency: Which word needs to be changed theleast to affect the score the most?

[Chatfield, Simonyan, Vedaldi, Zisserman, 2014][Denil, Demiraj, Kalchbrenner, Blunsom & NdF, 2014]

Inspiration: Visualisation & saliency[Chatfield, Simonyan, Vedaldi, Zisserman, 2014]

Review Sentiment-BasedSummarisation[Denil, Demiraj, Kalchbrenner, Blunsom & NdF, 2014]

Summarisation examples

Deep Multi-Instance Learning
Transfer
Breaking Privacy[Kotzias, Denil & NdF, 2014]

Deep Multi-Instance Learning[Kotzias, Denil & NdF, 2014]

Deep Multi-Instance Learning[Kotzias, Denil & NdF, 2014]

Deepmind approachDeepmind approachDeepmind approachImitation learning & mirror neurons

Imitation learning for Atari[Dejan, Miroslav, 2014]

Imitation learning for Atari[Dejan Markovikj, Miroslav Bogdanovic, Misha Denil, NdF 2014]

Attending for decision making

Highlights
1. Map objects (images, sentences, DNA sequences, voice commands, orcombinations of these) to points in metric spaces (embeddings).
2. Combinatorial optimization becomes easier in metric spaces.
3. Decode the embeddings to produce structured outputs. Think of parsing,sequential decision making and translation.
4. Distributed representations have exponential power and further gains
arise from the use of layers (composition).
5. The more data and the bigger the models, the more impressive theresults (eco-statistical view and parallelism).
6. Questions: Embeddings and logic? Computational power? What aregood encoders, decoders and transformers? Memory?

Thank you