Deep Sequence Modeling
MIT 6.S191

Ava Soleimany
January 28, 2019

Given an image of a ball,
can you predict where it will go next?

Given an image of a ball,
can you predict where it will go next?

???

Given an image of a ball,
can you predict where it will go next?

Given an image of a ball,
can you predict where it will go next?

Sequences in the wild

Audio
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19Sequences in the wild
character:

6.S191 Introduction to Deep Learning
word:

Text
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

A Sequence Modeling Problem:
Predict the Next WordA sequence modeling problem: predict the next word
“This morning I took my cat for a walk.”

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

A sequence modeling problem: predict the next word
“This morning I took my cat for a walk.”
given these words

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

A sequence modeling problem: predict the next word
“This morning I took my cat for a walk.”
given these words

predict thenext word

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Idea #1: use a fixed window“This morning I took my cat for a walk.”
given these
two words

predict thenext word

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Idea #1: use a fixed window“This morning I took my cat for a walk.”
given these
two words

predict thenext word

One-hot feature encoding: tells us what each word is[1000001000]foraprediction
6.S191 Introduction to Deep Learning
introtodeeplearning.com

Adapted from H. Suresh, 6.S191 2018
1/28/19

Problem #1: can’t model long-term dependencies“France is where I grew up, but I now live in Boston. I speak fluent ___.”

J’aime 6.S191!
We need information from the distant past to accurately
predict the correct word.

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Idea #2: use entire sequence as set of counts“This morning I took my cat for a”“bag of words”[ 0 1 0 0 1 0 0 … 0 0 1 1 0 0 0 1]

prediction
Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Problem #2: counts don’t preserve order
The food was good, not bad at all.
vs.

The food was bad, not good at all.

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Idea #3: use a really big fixed window“This morning I took my cat for a walk.”
given thesewords

predict thenext word[10000000010010001000 00010 … ]morning

I

took

this

cat

prediction
Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Problem #3: no parameter sharing[10000000010010001000 00010 … ]thismorning

took

the

cat

Each of these inputs has a separate parameter:

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Problem #3: no parameter sharing[10000000010010001000 00010 … ]thismorning

took

the

cat

Each of these inputs has a separate parameter:[0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 … ]thismorning

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Problem #3: no parameter sharing[10000000010010001000 00010 … ]thismorning

took

the

cat

Each of these inputs has a separate parameter:[0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 … ]thismorning

Things we learn about the sequence won’t transfer ifthey appear elsewhere in the sequence.
Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19Sequence modeling: design criteria
To model sequences, we need to:1.

Handle variable-length sequences

2. Track long-term dependencies
3.

Maintain information about order4.

Share parameters across the sequence

RNN

Today: Recurrent Neural Networks (RNNs) as
an approach to sequence modeling problems
Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Recurrent Neural Networks (RNNs)

Standard feed-forward neural network#"!
One to One“Vanilla” neural network[1]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Recurrent neural networks: sequence modeling#"!
One to One“Vanilla” neural network

Many to One
Sentiment Classification[1]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Recurrent neural networks: sequence modeling#"!
One to One“Vanilla” neural network

Many to One
Sentiment Classification

Many to Many
Music Generation
6.S191 Lab!
[1]

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Recurrent neural networks: sequence modeling#"… and many otherarchitectures andapplications!
One to One“Vanilla” neural network

Many to One
Sentiment Classification

Many to Many
Music Generation
6.S191 Lab!
[1]

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

A standard “vanilla” neural network
output vector$#"

input vector!"
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

A recurrent neural network (RNN)
output vector

$#"

RNN

input vector

ℎ"

!"
[2]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

A recurrent neural network (RNN)
output vector

$#"

RNN
recurrent cell

input vectorℎ"!"6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

A recurrent neural network (RNN)
output vector

RNN
recurrent cell

input vector

Apply a recurrence relation at every
time step to process a sequence:

$#"

ℎ"

!"
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

A recurrent neural network (RNN)
output vector

RNN
recurrent cell

input vector

Apply a recurrence relation at every
time step to process a sequence:-,"ℎ" = $% (ℎ"'( , *" )ℎ"cell statefunctionold stateparameterized
by W

input vector attime step t*"
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

A recurrent neural network (RNN)
output vector

RNN
recurrent cell

input vector

Apply a recurrence relation at every
time step to process a sequence:

-,"

*"

ℎ" = $% (ℎ"'( , *" )ℎ"new statefunctionold stateparameterized
by W

input vector attime step t

Note: the same function and set ofparameters are used at every time step

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNN state update and output
output vector

$#"

RNN
recurrent cell

input vectorℎ"!"6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNN state update and output
output vector

$#"

RNN
recurrent cell

input vectorℎ"!"Input Vector
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNN state update and output
output vector

21"
Update Hidden State

RNN
recurrent cell

input vectorℎ"ℎ" = tanh()** ℎ"+, + ).* /" )/"Input Vector
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNN state update and output
output vector

Output Vector

21"21" = )*3 ℎ"Update Hidden State

RNN
recurrent cell

input vectorℎ"ℎ" = tanh()** ℎ"+, + ).* /" )/"Input Vector
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNNs: computational graph across time

$#"

RNN

=

Represent as computational graph unrolled across time!"
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNNs: computational graph across time

$#"

RNN

!"

$#%

=
!%
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNNs: computational graph across time

$#"

RNN

!"

$#%

$#&

!%

!&

=

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNNs: computational graph across time

$#"

RNN

!"

$#%

$#&

$#'

…

$#"

!%

!&

!'

…

!"

=

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNNs: computational graph across time

$#"

RNN

$#%

$#'

…

$#"

=
()*

!"

$#&

!%

()*

()*

!&
6.S191 Introduction to Deep Learning
introtodeeplearning.com

!'

…

()*

!"
1/28/19

RNNs: computational graph across time

$#"

RNN

$#%

=

(**
()*

!"

$#&

!%

$#'

(**
()*

$#"

(**
()*

!&
6.S191 Introduction to Deep Learning
introtodeeplearning.com

…

!'

…

()*

!"
1/28/19

RNNs: computational graph across time

$#"

$#%

$#&

(*+

RNN

=

!"

(*+

(**
()*

!%

$#'

…

(*+

(**
()*

(*+

(**
()*

!&
6.S191 Introduction to Deep Learning
introtodeeplearning.com

$#"

!'

…

()*

!"
1/28/19

RNNs: computational graph across time
Re-use the same weight matrices at every time step
$#"

$#%

$#&

(*+

RNN

=

!"

(*+

(**
()*

!%

$#'

…

(*+

(**
()*

(*+

(**
()*

!&
6.S191 Introduction to Deep Learning
introtodeeplearning.com

$#"

!'

…

()*

!"
1/28/19

RNNs: computational graph across time
Forward pass

$#"

(%

(&

('

$#%

$#&

$#'

*,-

RNN

=

*,,
*+,

!"

*,-

!%

()

…

*,-

*,,
*+,

*,-

*,,
*+,

!&
6.S191 Introduction to Deep Learning
introtodeeplearning.com

$#"

!'

…

*+,

!"
1/28/19

RNNs: computational graph across time
Forward pass

$#"

(
(%

(&

('

$#%

$#&

$#'

*,-

RNN

=

*,,
*+,

!"

*,-

!%

()

…

*,-

*,,
*+,

*,-

*,,
*+,

!&
6.S191 Introduction to Deep Learning
introtodeeplearning.com

$#"

!'

…

*+,

!"
1/28/19

Backpropagation Through Time (BPTT)Recall: backpropagation in feed forward models"
Backpropagation algorithm:
1. Take the derivative (gradient) of theloss with respect to each parameter
2. Shift parameters in order tominimize loss!

[3]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNNs: backpropagation through time
Forward pass

$#"

(
(%

(&

('

$#%

$#&

$#'

*,-

RNN

=

*,,
*+,

!"

*,-

!%

()

…

*,-

*,,
*+,

*,-

*,,
*+,

!&
6.S191 Introduction to Deep Learning
introtodeeplearning.com

$#"

!'

…

*+,

!"
1/28/19

RNNs: backpropagation through time
Forward pass
Backward pass

$#"

(
(%

(&

('

$#%

$#&

$#'

*,-

RNN

=

*,,
*+,

!"

*,-

!%

()

…

*,-

*,,
*+,

$#"
*,-

*,,
*+,

!&

!'

…

*+,

!"
[4]

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Standard RNN gradient flow
ℎ"
'()

#"

'))

'))
'()

#$

'))
'()

#%

…

ℎ&
'()

#&

[1]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Standard RNN gradient flow
ℎ"
'()

#"

'))

'))
'()

#$

'))
'()

#%

…

ℎ&
'()

#&

Computing the gradient wrt ℎ" involves many factors of *++ (and repeated ,′!)

[1]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Standard RNN gradient flow: exploding gradients
ℎ"
'()

'))

'))
'()

#"

#$

'))
'()

#%

…

ℎ&
'()

#&

Computing the gradient wrt ℎ" involves many factors of *++ (and repeated ,′!)
Many values > 1:
exploding gradients

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Standard RNN gradient flow: exploding gradients
ℎ"
'()

'))

'))
'()

#"

#$

'))
'()

#%

…

ℎ&
'()

#&

Computing the gradient wrt ℎ" involves many factors of *++ (and repeated ,′!)
Many values > 1:
exploding gradients
Gradient clipping toscale big gradients[1]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Standard RNN gradient flow: vanishing gradients
ℎ"
'()

'))

'))
'()

#"

#$

'))
'()

#%

…

ℎ&
'()

#&

Computing the gradient wrt ℎ" involves many factors of *++ (and repeated ,′!)
Many values < 1:
vanishing gradients

Many values > 1:
exploding gradients
Gradient clipping toscale big gradients[1]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Standard RNN gradient flow: vanishing gradients
ℎ"
'()

'))

'))
'()

#"

#$

'))
'()

#%

…

ℎ&
'()

#&

Computing the gradient wrt ℎ" involves many factors of *++ (and repeated ,′!)
Largest singular value > 1:
exploding gradients

Largest singular value < 1:
vanishing gradients

Gradient clipping toscale big gradients

1. Activation function
2. Weight initialization
3. Network architecture[1]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The problem of long-term dependencies
Why are vanishing gradients a problem?

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The problem of long-term dependencies
Why are vanishing gradients a problem?
Multiply many small numbers together

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The problem of long-term dependencies
Why are vanishing gradients a problem?
Multiply many small numbers together

Errors due to further back time steps
have smaller and smaller gradients

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The problem of long-term dependencies
Why are vanishing gradients a problem?
Multiply many small numbers together

Errors due to further back time steps
have smaller and smaller gradients

Bias network to capture short-term
dependencies

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The problem of long-term dependencies
Why are vanishing gradients a problem?

“The clouds are in the ___”

Multiply many small numbers together

Errors due to further back time steps
have smaller and smaller gradients

Bias network to capture short-term
dependencies

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The problem of long-term dependencies
Why are vanishing gradients a problem?

“The clouds are in the ___”
$#"

$#%

$#&

$#'

$#(

!"

!%

!&

!'

!(

Multiply many small numbers together

Errors due to further back time steps
have smaller and smaller gradients

Bias parameters to capture short-term
dependencies

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The problem of long-term dependencies
Why are vanishing gradients a problem?

“The clouds are in the ___”
$#"

$#%

$#&

$#'

$#(

!"

!%

!&

!'

!(

Multiply many small numbers together

Errors due to further back time steps
have smaller and smaller gradients“I grew up in France, … and I l speak fluent___ ”

Bias parameters to capture short-term
dependencies

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The problem of long-term dependencies
Why are vanishing gradients a problem?

“The clouds are in the ___”
$#"

$#%

$#&

$#'

$#(

!"

!%

!&

!'

!(

Multiply many small numbers together

Errors due to further back time steps
have smaller and smaller gradients“I grew up in France, … and I l speak fluent___ ”
$#"

$#%

…

$#)

$#)*%

!"

!%

…

!)

!)*%

Bias parameters to capture short-term
dependencies

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Trick #1: activation functions
ReLU derivative

Using ReLU prevents! " from shrinking thegradients when # > 0
tanh derivative
sigmoid derivative

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Trick #2: parameter initialization
Initialize weights to identity matrix
Initialize biases to zeroThis helps prevent the weights from shrinking to zero.
Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Solution #3: gated cells
Idea: use a more complex recurrent unit with gates tocontrol what information is passed throughgated cell
LSTM, GRU, etc.

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Solution #3: gated cells
Idea: use a more complex recurrent unit with gates tocontrol what information is passed throughgated cell
LSTM, GRU, etc.

Long Short Term Memory (LSTMs) networks rely on a gated cell to
track information throughout many time steps.
Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Long Short Term Memory (LSTM) Networks

Standard RNN
In a standard RNN, repeating modules contain a simple computation node
ℎ"

ℎ"&%

tanh

!"&%

ℎ"$%

tanh

!"

tanh

!"$%

[2]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Long Short Term Memory (LSTMs)
LSTM repeating modules contain interacting layers that control information flow
ℎ"

ℎ")*

tanh

$ $ tanh

!")*

$

ℎ"+*

tanh

tanh

$ $ tanh$!"$ $ tanh$!"+*

LSTM cells are able to track information throughout many timesteps
Hochreiter & Schmidhuber, 1997 [2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Long Short Term Memory (LSTMs)
LSTMs maintain a cell state !" where it’s easy for information to flow
ℎ"

!"

!"$%
tanh

ℎ"$%

'

'

tanh

'

ℎ"

&"
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Long Short Term Memory (LSTMs)
Information is added or removed to cell state through structures called gates!
Gates optionally let information through, via a sigmoid
neural net layer and pointwise multiplication[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Long Short Term Memory (LSTMs)
How do LSTMs work?
ℎ"

("

(")*
tanh

,"
ℎ")*

#

#

tanh

#

ℎ"

+"
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Long Short Term Memory (LSTMs)
LSTMs forget irrelevant parts of the previous state
ℎ"

("

(")*
tanh

,"
ℎ")*

#

#

tanh

#

ℎ"

+"
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Long Short Term Memory (LSTMs)
LSTMs selectively update cell state values
ℎ"

+"

+"#$
tanh

ℎ"#$

&

&

tanh

&

ℎ"

%"
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Long Short Term Memory (LSTMs)
LSTMs use an output gate to output certain parts of the cell state
ℎ"

("

(")*
tanh

,"
ℎ")*

#

#

tanh

#

ℎ"

+"
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Long Short Term Memory (LSTMs)
How do LSTMs work?
1) Forget 2) Update 3) Output
ℎ"

("

(")*
tanh

,"
ℎ")*

#

#

tanh

#

ℎ"

+"
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

LSTMs: forget irrelevant information
ℎ"

("

(")*
tanh

,"
ℎ")*

#

#

tanh

#

," = .(01 ℎ")* , +" + 45 )- Use previous cell output and input- Sigmoid: value 0 and 1 – “completely
forget” vs. “completely keep”ℎ"+"ex: Forget the gender pronoun of previous subject in sentence.
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

LSTMs: identify new information to be stored
ℎ"

!"#$

!"
,"

ℎ"#$

&

&

tanh

.-"
tanh

," = 0(23 ℎ"#$ , +" + 67 ).-" = tanh(29 ℎ"#$ , +" + 6: )- Sigmoid layer: decide what values to update

&

ℎ"

+"

- Tanh layer: generate new vector of“candidate values” that could be added tothe state

ex: Add gender of new subject to replace that of old subject.
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

LSTMs: update cell state
ℎ"

+"

+")*
,"

/"
ℎ")*

#

#

tanh

.-"
tanh

#

ℎ"

." = /" ∗ .")* + ," ∗ .-"- Apply forget operation to previous
internal cell state: /" ∗ .")*
- Add new candidate values, scaled by how
much we decided to update: ," ∗ .-"("ex: Actually drop old information and add new information about subject’s gender.
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

LSTMs: output filtered version of cell state
ℎ"

("

(")*

," = 3(45 ℎ")* , +" + 89 )ℎ" = ," ∗ tanh(0" )

tanh

,"
ℎ")*

#

#

tanh

#

ℎ"

+"

- Sigmoid layer: decide what parts of state tooutput- Tanh layer: squash values between -1 and 1- ," ∗ tanh(0" ): output filtered version of cellstate

ex: Having seen a subject, may output information relating to a verb.
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

LSTM gradient flow
Backpropagation from !" to !"#$ requires only elementwise multiplication!
No matrix multiplication à avoid vanishing gradient problem.
ℎ"!" = ." ∗ !"#$ + 1" ∗ !2"

,"

,"#$
tanh

ℎ"#$

'

'

tanh

'

ℎ"

&"
[2, 5]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

LSTM gradient flow
Uninterrupted gradient flow!
ℎ"

ℎ)

+,

+)

+"

tanh

$

!)

$

tanh

$

ℎ*

+*

tanh

$

$

tanh

$

!"

tanh

$

$

tanh

$

!*

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

LSTMs: key concepts
1. Maintain a separate cell state from what is outputted
2. Use gates to control the flow of information• Forget gate gets rid of irrelevant information• Selectively update cell state• Output gate returns a filtered version of the cell state

3. Backpropagation from !" to !#$% doesn’t require matrix multiplication:
uninterrupted gradient flow

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

RNN Applications

Example task: music generation
F#

G

C

Input: sheet music

A

Output: next character in sheet music

E

F#

G

C
6.S191 Lab!

Adapted from H. Suresh, 6.S191 2018

[6]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Example task: sentiment classification
sentiment<positive>

Input:

sequence of words

Output:probability of having positive sentiment

loss = tf.nn.softmax_cross_entropy_with_logits(labels=model.y, logits=model.pred
)

I

love

this

class!

Adapted from H. Suresh, 6.S191 2018[7]
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Example task: sentiment classification
Tweet sentiment classification

sentiment<positive>

I

love

this

class!

Adapted from H. Suresh, 6.S191 2018
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Example task: machine translation

theAdapted from H. Suresh, 6.S191 2018

dog

eats

le

chien

mange

<start>lechien

Encoder (English)

Decoder (French)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[8,9]
1/28/19

Example task: machine translation
le

chien

mange

<start>lechien

encodingbottleneckthe

Adapted from H. Suresh, 6.S191 2018

dog

eats

Encoder (English)

Decoder (French)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[8,9]
1/28/19

Attention mechanisms
le

chien

mange

<start>lechien

Attention mechanisms in neural networks
provide learnable memory accesstheAdapted from H. Suresh, 6.S191 2018

dog

eats

Encoder (English)

Decoder (French)
6.S191 Introduction to Deep Learning
introtodeeplearning.com[8,9]
1/28/19

Recurrent neural networks (RNNs)
1. RNNs are well suited for sequence modeling tasks
2. Model sequences via a recurrence relation
3. Training RNNs with backpropagation through time
4. Gated cells like LSTMs let us model long-term dependencies
5. Models for music generation, classification, machine translation

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19References:
goo.gl/hbLkF6

6.S191: Introduction to Deep Learning
Lab 1: Introduction to Tensorflow and Music Generation with RNNs
Link to download labs:
http://introtodeeplearning.com#schedule
1. Open the lab in Google Colab
2. Start executing code blocks and filling in the #TODOs
3. Need help? Find a TA or come to the front!!

End of Slides