CSE 587: Deep Learning for Natural
Language Processing
Lecture 5. Transformers

Rui Zhang
Spring 2023

1

Outline
NLP
Contextualized Word Embeddings
Subword Tokenization
Byte-Pair Encoding
GPT, BERT, RoBERTa
MLTransformer
Self Attention

2

Recall: RNN Encoder-Decoder; Attention; Self Attention

3

Recall: RNN Encoder-Decoder; Attention; Self Attention

4

Recall: RNN Encoder-Decoder; Attention; Self Attention

5

Attention Is All You Need (Vaswani et al., 2017)
Transformer

6

Transformer - Key Idea
In RNN, the first token is linked to next token, and so on, using RNN unit.

In Transformer, every token is linked to all other tokens using self attention.

https://d2l.ai/chapter_attention-mechanisms/self-attenti
on-and-positional-encoding.html

7

Transformer - Architecture Overview

8Transformer - Architecture Overview
Encoder9Transformer - Architecture Overview
Decoder
Encoder10Transformer - Self Attention

11

Transformer - Self Attention

Figures from Emma Strubell

12

Transformer - Self Attention

Figures from Emma Strubell

13

Transformer - Self Attention
Q: Query
K: Key
V: Value

Figures from Emma Strubell

14

Transformer - Self Attention
Q: Query
K: Key
V: Value

Figures from Emma Strubell

15

Transformer - Self Attention
Calculate Attention scores: between Q: Query and K: Key

Figures from Emma Strubell

16

Transformer - Self Attention
Each word is attended to all other words.

Figures from Emma Strubell

17

Transformer - Self Attention

Figures from Emma Strubell

18

Transformer - Self Attention

Figures from Emma Strubell

19

Transformer - Multi-Head Self Attention

Figures from Emma Strubell

20

Transformer - Multi-Head Self Attention

21

Transformer - Multi-Head Self Attention

https://jalammar.github.io/illustrated-transformer/
22

Transformer Layer

Figures from Emma Strubell

23

Transformer

Figures from Emma Strubell

24

Contextualized Word Embeddings●●

Problems with word2vec/GloVe: single representation for each word.
Contextualized Word Embeddings: representations depend on context.

Figures from Emma Strubell

25

Other Components in Transformers
●
●
●
●
●

Residual Connection
Layer Normalization
Masked Self Attention in Decoder
Encoder-Decoder Cross Attention
Sinusoidal Positional Encoding

26

Residual ConnectionResidual Connection (He et al., 2016)● Add input to its output

27

Layer Normalization
Layer Normalization (Ba et al., 2016)28

Masked Self Attention in Decoder●●

In encoder, we can always look at all input words.
In decoder, we can only look at previous words.
○○

That means, we need to "mask" future words when
performing attention.
Words are blocked for attending to future words during
training.

29

Encoder-Decoder Cross Attention●Attention between encoder self-attention outputs
and decoder masked self attention outputs.

30Positional Encoding●●●

There is no recurrence (RNN) and no convolution (CNN).
Self-attention do not have order information.
So we need positional encoding to bring position information.

31

Sinusoidal Positional Encoding●●

There are many different choices of positional encodings (Gehring et al., 2017)○ some are learned and some are fixed○ some for relative position and some for absolute position.
In Transformer, we use Sinusoidal position encodings: fixed embeddings and cangeneralize to any sequence length.

32

Sinusoidal Positional Encoding●●●

There are many different choices of positional encodings (Gehring et al., 2017)○ some are learned and some are fixed○ some for relative position and some for absolute position.
In Transformer, we use Sinusoidal position encodings: fixed embeddings and cangeneralize to any sequence length.
Intuition: (4-dimensional position encodings for 16 positions)

https://kazemnejad.com/blog/transformer_architecture_positional_encoding/33

Sinusoidal Positional Encoding

4-dimensional position
encodings for 16 positions

https://kazemnejad.com/blog/transformer_architecture_positional_encoding/34

Sinusoidal Positional Encoding

t: position index
i: dimension index
d: total dimension

https://kazemnejad.com/blog/transformer_architecture_positional_encoding/35

Sinusoidal Positional Encoding - Properties●●●●

It should output a unique encoding for each time-step (word’s position in asentence)
Distance between any two time-steps should be consistent across sentenceswith different lengths.
Our model should generalize to longer sentences without any efforts. Itsvalues should be bounded.
It must be deterministic.

https://kazemnejad.com/blog/transformer_architecture_positional_encoding/36

Other Tricks in Transformers●●●

Subword Tokenization
Optimizer
Label Smoothing

37

OOV Issue
How to handle unknown words, i.e., a word at test time that we’ve never seen in
our training data. It is also called OOV (out-of-vocabulary) words.
If we use word-level tokenization, we cannot have an index for unknown words,
and we cannot have a word embedding for unknown words.
Old solution: replace low-frequency words in training data with a special token<UNK>; then after training, we have a word embedding for <UNK>; if we have an
unknown word during testing, we replace it with <UNK>.
New solution: Subword tokenization.
38

Subword Tokenization
Use Subword Units for tokenization.
Based on a simple algorithm called byte pair encoding (Gage, 1994).
First used successfully for NLP for machine translation by (Sennrich et al., 2016).

39

Byte Pair Encoding
Perform pre-tokenization

Form a base vocabulary: b, g, h, n, p, s, u
Slides Credit: Mohit Iyer

https://huggingface.co/transformers/tokenizer_summary.html

40

Byte Pair EncodingCount all the character pair.

Slides Credit: Mohit Iyer

https://huggingface.co/transformers/tokenizer_summary.html

41

Byte Pair Encoding
Merge the most frequent character pair, i.e., ug, and add it to the vocabulary.
Then rotokenize and repeat this process.
The next to merge is un, so we add un to the vocabulary.

Slides Credit: Mohit Iyer

https://huggingface.co/transformers/tokenizer_summary.html

42

Byte Pair Encoding
Repeat until some fixed number of merges, or until we reach a target vocab size.

new vocab: b, g, h, n, p, s, u, ug, un, hug
Slides Credit: Mohit Iyer

https://huggingface.co/transformers/tokenizer_summary.html

43

Byte Pair Encoding
To avoid OOV, all possible characters / symbols need to be included in the basevocab.
● This can be huge if including all unicode characters (size 144,697)!
● GPT-2 uses bytes as the base vocabulary (size 256) and then applies BPE
on top of this sequence (with some rules to prevent certain types of merges).
● GPT-2 has a vocabulary size of 50,257, which includes 256 bytes basetokens, a special end-of-text token, and the symbols learned with 50,000
merges.
Commonly have vocabulary sizes of 32K to 64K

Slides Credit: Mohit Iyer

44

Encoding and Decoding
Encoding● Sort the vocab from longest subwords to shortest subwords.
● Go through the vocab, and replace the longest subword first.
● This is very computationally expensive.
● In practice, we can cache how popular words are encoded.
Decoding● In fact, we will add </w> to each word to indicate word boundaries.
● For example, if the encoded sequence is [“the</w>”, “high”, “est</w>”,“moun”, “tain</w>”], we immediately know the decoded sequence “the highestmountain</w>”.
45

Other subword Tokenization
Byte Pair Encoding●●

merge by frequency
used by GPT-2.

WordPiece (Schuster et al., 2012)●●

merge by likelihood given by language models, not by frequency
used by BERT.

SentencePiece (Kudo et al., 2018)●●●

All tokenization algorithms described so far have the same problem: It is assumed thatthe input text uses spaces to separate words. However, not all languages use spaces toseparate words.
SentencePiece can train subword models directly from raw sentences.
used in HW2.
46

Can we don't do tokenization at all?
Old solution: use <UNK>.
New solution: Subword tokenization
New New solution: tokenizer-free and directly operate on bytes!
● CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language
Representation.
● ByT5: Towards a token-free future with pre-trained byte-to-byte models.

47

Optimizer
Adam Optimizer with Linear Warmup Learning Rate48

Label Smoothinghttps://towardsdatascience.com/label-smoothing-make-your-model-less-over-confident-b12ea6f81a9a49

Computational Complexity: Quadratic in Length!

50

Results

51

Model Variations

52

Transformer-based Language Models
It is becoming the underlying architecture of most popular language models. Twomost famous are:
● GPT: OpenAI Transformer-based Language Models● BERT: Google Transformer-based Language Models● RoBERTa: Facebook Transformer-based Language Models

53

Transformer beyond NLP
Transformer can also be used in many other applications: Vision, Speech,
Reinforcement Learning, ...
People even consider we no longer needs CNN/RNN or any other neural nets,and Transformer is unifying many ML problems.
● Check Facebook's data2vec● Check this Twitter post by Andrej Karpathy"But as of approx. last two years, even the neural net architectures across allareas are starting to look identical - a Transformer (deﬁnable in ~200 lines ofPyTorch github.com/karpathy/minGP …), with very minor differences. Either as astrong baseline or (often) state of the art."
https://T/blob/master/mingpt/model.py

54

Reading
Annotated Transformers by Alexander M. RushThe Illustrated Transformer by Jay Alammar

55