CSE 587: Deep Learning for NaturalLanguage Processing
Lecture 4. RNNs, Sequence-to-Sequence, Attention

Rui Zhang
Spring 2023

1

Outline - Key Concepts
NLP
Sequence-to-Sequence Tasks
Sentence Representation
ML
Recurrent Neural Networks
Teacher Forcing
Greedy Decoding
Attention2

Natural Language is Sequential
Natural Language is Sequential● words are sequences of characters.
● sentences are sequences of words.
● paragraphs/documents/dialogues are sequences of sentences.

3

Natural Language is Sequential
We need to model the order and dependency in sequential data!
The Long-Distance Dependency problem:
What is the referent of "they"?
●

The city councilmen refused the demonstrators a permit because they fearedviolence.

●

The city councilmen refused the demonstrators a permit because they
advocated violence.

(from Winograd Schema Challenge: http://commonsensereasoning.org/winograd.html)
4Natural Language is Sequential
We need to model the order and dependency in sequential data!
The Long-Distance Dependency problem:
What is the referent of "they"?
●

The city councilmen refused the demonstrators a permit because they fearedviolence.

●The city councilmen refused the demonstrators a permit because they
advocated violence.

(from Winograd Schema Challenge: http://commonsensereasoning.org/winograd.html)
5

Recurrent Neural Networks

https://colah.github.io/posts/2015-08-Understanding-LSTMs/

6

Recurrent Neural Networks - Equation7https://people.cs.umass.edu/~miyyer/cs685/slides/02-neural-lms.pdf

8Parameter Tying

Slides Credit: Graham Neubig

9RNN Variants
Bidirectional RNN

Multilayer RNN

Long Short Term Memory (LSTM)

Gated Recurrent Unit (GRU)

10Bidirectional RNN
One RNN from left to right; The other RNN from right to left.

11Multilayer RNN / Deep RNN
Adding another layer of RNNs on top of the outputs of lower-layer RNNs.

https://d2l.ai/chapter_recurrent-modern/deep-rnn.html

12Long Short Term Memory (LSTM)
Use several "gates" to control adding or removing information.

https://colah.github.io/posts/2015-08-Understanding-LSTMs/
13

RNNs as Sentence Encoders
RNNs can be used to Encode Sentence, i.e., we can get an representation of thesentence.
● You can use the last hidden state as the representation of the sentence.
● You can use the average of all the hidden states as the representation of thesentence.

14

Sentence Representation
Then, you can use the the sentence representation for● Sentence Classification● Paraphrase Identification● Semantic Similarity/Relatedness● Entailment● Retrieval and Ranking

15Semantic Similarity/Relatedness●●●SICK (Sentences Involving Compositional Knowledge) Dataset (Marelli et al.
2014).
The relatedness score ranges from 1 to 5.
Hugging Face dataset viewer for SICK.

16

Textual Entailment
Entailment: if A is true, then B is true
Contradiction: if A is true, then B is not true
Neutral: cannot say either of the above
e.g., The Stanford Natural Language Inference (SNLI) Corpus

17Output can be a sequence too
Input X
Text
Text
Text

Output Y

Task

Label
Linguistic Structure
Text

Text Classification (e.g., Sentiment Analysis)
Structured Prediction (e.g., Part-of-Speech Tagging)
Text Generation (e.g., Translation, Summarization)

Sentiment Analysis●I really like this movie. -> Positive, Neutral, NegativePart-of-Speech Tagging:●

Time flies like an arrow. -> N V IN DET N

Machine Translation●Time flies like an arrow. -> 时间飞逝如箭。

18

Sequence-to-Sequence●●●●Input is sequence and output is also a sequence
Use an encoder to encode input, and an decoder to decode output.
So it is also known as Encoder-Decoder Model.
Many problems can be casted as a sequence-to-sequence learning task.

Sutskever et al. 201419

Training: Teacher Forcing●●●●

During training, we use the original output sequence (token labels) is fed into the decoder.
This is called Teacher Forcing.
Suppose your training data has one example of (ABC<EOS>, WXYZ<EOS>).
Calculate the loss for five decoding time steps, and add them together as the final loss function○ ABC<EOS>
W○ ABC<EOS>W
X○ ABC<EOS>WXY○ ABC<EOS>WXYZ○ ABC<EOS>WXYZ
<EOS>

20

https://people.cs.umass.edu/~miyyer/cs685/slides/05-transformers.pdf

21Prediction●After the model is trained, we run inferenceor prediction on test and dev set.

●During prediction, we need to use thepredicted token from the previous time
step as the current input to the decoder.

https://towardsdatascience.com/what-is-teacher-forcing-3da6217fed1c22

https://people.cs.umass.edu/~miyyer/cs685/slides/05-tr
ansformers.pdf

23Decoding: Greedy (Beam Search with Size = 1)●●

There are different ways of decoding (we will talk about this more in NLG.)
The simplest decoding algorithm is greedy, i.e., beam search with size=1.

https://lorenlugosch.github.io/posts/2019/02/seq2seq/

24Sequence-to-Sequence Applications
Many problems can be casted as sequence-to-sequence learning tasks.
Input

OutputTask

Structured DataNL Description
Data-to-Text Generation
Source Language
Target Language
Machine Translation
Long Document
Short Summary
Summarization
Question
Structured Meaning Representation
Semantic Parsing
Dialog Utterance
Response
Dialogue Response Generation
Image
Text
Image Captioning
SpeechTranscript
Speech Recognition25

Sentence Representation from Encoder
We only feed the last hidden state of encoder to the decoder.
This means the meaning of the whole sentence is loaded into the single vector.
Can use multiple vectors from the encoder during decoding?

https://www.guru99.com/seq2seq-model.html

26

Attention
In Machine Translation, at each step of decoding, thedecoder should focus on different parts of the words,
with different amounts of "attentions".
● Each hidden state of the encoder is arepresentation of a input word.
● The decoder will look at all the encoder hiddenstates.
● It computes "attention weights", and use this toperform a linear combination of encoder hiddenstates.

Bahdanau et al. 201527

Calculate Attention●●●Use “query” vector (decoder state) and “key” vectors (all encoder states)
For each query-key pair, calculate weight
Normalize to add to one using softmax

Slides Credit: Graham Neubig

28

Calculate Attention●●Combine together value vectors (usually encoder states, like key vectors) bytaking the weighted sum.
Use this in any part of the model you like, e.g., predicting the next word.

Slides Credit: Graham Neubig

29

Attention Visualization

Bahdanau et al. 201530

Attention Score Functions
x in the encoder hidden state (key), h is the decoder hidden state (query)
Additive Multilayer Perceptron / Feedforward Neural Network (Bahdanau et al.
2015)

Bilinear (Luong et al. 2015)31

Attention Score Functions
x in the encoder hidden state (key), h is the decoder hidden state (query)
Dot Product (Luong et al. 2015)

Scaled Dot Product (Vaswani et al. 2017)32

Self Attention
Attention within the encoder itself.

Cheng et al., 201633

Attention in Image Captioning
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

34

Attention in Speech Recognition
Listen, Attend and Spell

35