CSE 587: Deep Learning for Natural
Language Processing
Lecture 6. BERT and Pretraining

Rui Zhang
Spring 2023

1

Outline
NLP
Pretrained Language Models
BERT and its variants
Model Analysis
ML
Pretraining
Finetuning2

Transformer

3How is Transformer used
We can use Transformer separately for each task.
Transformer is initialized randomly, and trained for each dataset using supervised
learning.

Sentiment Analysis

SA Data

Vanilla Transformer

Machine Translation

MT Data

Vanilla Transformer

Named Entity Recognition

NER Data

Vanilla Transformer

4Pretraining●●●

First train Transformer using a lot of general text using unsupervised learning.
This is called pretraining.
Then train the pretrained Transformer for a specific task using supervised
learning.This is called finetuning.
The whole process can be called transfer learning.

Large Corpus

Transformer

Pretraining

Sentiment Analysis

Finetuning

Machine Translation

Finetuning

Named Entity Recognition

Finetuning5

Slides Credit: Colin Raffel

6

Slides Credit: Colin Raffel

7

BERT (Devlin et al. 2018)
Model: Only use Transformer Encoder (no decoder part)
Data: BooksCorpus (800 million words) + English Wikipedia (2,500 million words)
Training Objective● Masked Language Modeling: predict word given bidirectional context.
● Next-sentence Prediction: predict the next sentence given the current
sentence.

8

BERT Input

9Training Objective 1: Masked Language Modelinghttps://jalammar.github.io/illustrated-bert/

10

Training Objective 1: Masked Language Modeling
Predict a random 15% of (sub)word tokens, and of these 15%:
● 80%: Replace input word with [MASK]● 10%: Replace input word with a random token● 10%: Leave input word unchanged 10% (but still predict it!)11

Training Objective 2: Next-sentence Prediction
Give two sentences as input, classify if the second sentence really follows the first one.

https://jalammar.github.io/illustrated-bert/

12

BERT on Different Tasks
We can use BERT for different tasks by changing the inputs and addingclassification layers on top of output embeddings.

"We show that pre-trained representations reduce the need for manyheavily-engineered task specific architectures. BERT is the first finetuning
based representation model that achieves state-of-the-art performance on alarge suite of sentence-level and token-level tasks, outperforming manytask-specific architectures."

13

BERT on Different Tasks
Sentence Classification

14

BERT on Different Tasks
Sentence Pair Classification

15

BERT on Different Tasks
Sequence Labeling

16

BERT on Different Tasks
Question Answering

17

BERT
Initially two BERT models are trained and released with the paper● BERT-base: 12 layers, 768-dim hidden states, 12 attention heads, 110 million
params.
● BERT-large: 24 layers, 1024-dim hidden states, 16 attention heads, 340
million params.
Pretraining is expensive● 64 TPU chips for a total of 4 days

18

What Does BERT Learns?
1. Semantic Knowledge
to tip a ___ : waiter > chef > robin
2. Syntactic Knowledge
3. World Knowledge

A Primer in BERTology: What we know about how BERT works (Rogers et al.,
2020)

19

Why Pretraining Works? Is BERT just memorizing?
Size of Data (~15GB)● BookCorpus (4.6 GB)● English Wikipeida (10 GB)

Size of Model● bert-base (110M x 4 bytes = 0.44 GB)● bert-large (340M x 4 bytes = 1.36 GB)
Not just memorize, but also compress

20How much does it cost?
The Cost of Training NLP Models: A Concise Overview (Sharir et al., 2020)
1. The cost of one training run
2. A typical fully-loaded cost with hyper-parameter tuning and multiple runs persetting21

How to Train BERT with an Academic Budget (Izsak et al., 2021)
24-hours, $50~$100
Our recipe consists of many different components thattogether amount to very large speedups:● Short sequences (Devlin et al., 2019)● Single-sequence training (Joshi et al., 2020)● Training larger models (Li et al., 2020b)● DeepSpeed (Rasley et al., 2020)● Sparse token prediction (Liu et al., 2019)● Fused implementations● Avoiding disk I/O● Large batch sizes (Liu et al., 2019)● Large learning rates (Liu et al., 2019)● Short warmup● Synchronizing schedule with time budget (Li et al.,2020a)
22

Pretraining for three types of architectures

Slides from John Hewitt

23

RoBERTa (Liu et al. 2019)
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Model: same as BERT
Data: same as bert
Training Objective● MLM same as BERT, but train longer● Remove next-sentence prediction.
Takeaway: more compute and more data can help; next-sentence prediction notnecessary.

24

SpanBERT (Joshi et al., 2019)
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Model: same as BERT
Data: same as bert
Training Objective● Masking contiguous random spans, rather than random tokens● Training the span boundary representations to predict the entire content of the masked span,
without relying on the individual token representations within it.
Takeaway: predicting entire spans is better than random tokens

25

GPT (Radford et al., 2018)
GPT● Generative Pretrained Transformer● Generative PreTraining
Model: only Transformer decoder.
Data: BooksCorpus: over 7000 unique books.
Training Objective: Language Modeling
Followed by GPT-2 and GPT-3● GPT (Jun 2018): 117 million parameters● GPT-2 (Feb 2019): 1.5 billion parameters● GPT-3 (July 2020): 175 billion parameters
26

ELECTRA (Clark et al. 2020)
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Model: Same as BERT
Data: Same as BERT
Training Objective:

27

ELECTRA (Clark et al. 2020)
the task is defined over all input tokens rather than just the small subset that was masked
out. As a result, the contextual representations learned by our approach substantiallyoutperform the ones learned by BERT given the same model size, data, and compute.
This makes training more efficient.

28

T5 (Raffel et al., 2019)
T5: “Text-to-Text Transfer Transformer”
Treat every text processing problem as a “text-to-text” problem, i.e. taking text as input andproducing new text as output.

29

T5 (Raffel et al., 2019)
Model: Transformer Encoder-Decoder
Data: C4 (Colossal Clean Crawled Corpus), clean English text scraped from the we (750GB).
Training Objective● Span Corruption (denoising)30

Span Corruption (denoising)31

T5 (Raffel et al., 2019)
T5 can be used for both classification (GLUE) and generation tasks such as
Summarization (CNNDM), Machine Translation (EnDe, EnFr, EnRo).

32

BART (Lewis et al., 2019)
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language
Generation, Translation, and Comprehension.

33

SciBERT (Beltagy et al., 2020)
SciBERT: A Pretrained Language Model for Scientific Text.
Keep pretraining BERT on in-domain data improves the end task performance.

34

REALM (Guu et al., 2020)
REALM: Retrieval-Augmented Language Model Pre-Training
These pre-trained models, such as BERT and RoBERTa, have been shown to memorize a surprising amount of world knowledge,
such as “the birthplace of Francesco Bartolomeo Conti”, “the developer of JDK” and “the owner of Border TV”. … these models
memorize knowledge implicitly — i.e., world knowledge is captured in an abstract way in the model weights …
Instead, what if there was a method for pre-training that could access knowledge explicitly, e.g., by referencing an additional large
external text corpus, in order to achieve accurate results without increasing the model size or complexity?

https://ai.googleblog.com/2020/08/realm-integrating-retrieval-into.html

35

REALM (Guu et al., 2020)
Open-Domain Question Answering

36

RAG (Lewis et al., 2020)
RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
REALM is mostly used for short-span QA, but RAG can be used for generation-based QA.

https://ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/37

RAG (Lewis et al., 2020)

38

ALBERT (Lan et al. 2019)
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Can we have a smaller model but with equal performance?
Two techniques to reduce the number of parameters.
● Factorized embedding: Decompose the large vocabulary embedding matrix
into two small matrices.
● Cross-layer parameter sharing.

39

DistilBERT (Sanh et al. 2019)
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Use knowledge distillation during the pre-training phase.
Knowledge distillation [Bucila et al., 2006, Hinton et al., 2015] is a compression
technique in which a compact model - the student - is trained to reproduce thebehaviour of a larger model - the teacher - or an ensemble of models.

40

Longformer (Beltagy et al., 2020)
Longformer: The Long-Document Transformer

41

Linformer (Wang et al., 2020)
Add low-rank linear projections for keys and values (but not for queries).
● k: projection dimension● n: input length● k << n42Transformer-XL (Dai et al., 2019)
Extend beyond a fixed lengths by using segments with recurrence.
XL means extra long.
Vanilla Transformer

Transformer-XL

43

XLNet (Yang et al. 2019)
Model: similar as BERT, but● Get rid of [MASK] token by masking attention to consider all possible
different orders of prediction (see next slide).
● Incorporate Transformer-XL to model longer sequence.
Data: Same as BERT + Giga5, ClueWeb 2012-B, Common Crawl
Training Objective● Permutation language modeling objective: sample a permutation of theoriginal sentence, and predict the permuted sentence in order (see next
slide).
● This combines the idea of autoregressive and autoencoding.
44

XLNet (Yang et al. 2019)45

Reformer (Kitaev et al., 2019)
●
●
●
●
●

During attention computation, we are interested in softmax(QKT).
Since softmax is dominated by the largest elements, we are only interested in nearest neighbors
during attention computation.
For example, if K is of length 64, for each qi we could only consider a small subset of, say, the 32 or
64 closest keys.
This is much more efficient, but how can we find the nearest neighbors among the keys?
Finding nearest neighbors quickly in high-dimensional spaces can be solved by locality-sensitive
hashing (LSH).

46

Reformer (Kitaev et al., 2019)●●

Replace dot-product attention by one that uses LSH.
Changing its complexity from O(L2) to O(L log L).

47

Performer (Choromanski et al., 2020)

48

Other Efficient Transformers●

Sparse Transformers (Child et al., 2019)○○●

Routing Transformers (Roy et al., 2020)
○
○
○

●

Introduce sparse factorizations of the attention matrix.
Reduce quadratic complexity to O(n √ n).
Content-based sparse attention.
Tokens are assigned to different clusters.
Attention is performed only within each cluster.

Nystromformer (Xiong et al., 2021)○

Approximate self-attention using the Nystrom method.

49

Efficient Transformers: A Survey (Tay et al., 2020)

50

Analysis on BERTBertViz is an interactive tool for visualizing attention in Transformer language models such as BERT,
GPT-2, or T5. Check out its code and demo here: https://github.com/jessevig/bertviz

51

Analysis on BERT
What Does BERT Look At? An Analysis of BERT’s Attention (Clark et al., 2019)52

Analysis on BERT
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, theRest Can Be Pruned (Voita et al., 2019)● Analyze Multi-head self-attention in Neural Machine Translation● Most important and confident heads play consistent and often
linguistically-interpretable roles.
● Can prune less important heads.

53

Analysis on BERT
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, theRest Can Be Pruned (Voita et al., 2019)● Heads have different "confidence", measured as average of its maximum
attention weight.

54

Analysis on BERT
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, theRest Can Be Pruned (Voita et al., 2019)● There are different types of heads.
● Positional heads: We refer to a head as “positional” if at least 90% of thetime its maximum attention weight is assigned to a specific relative position (inpractice either -1 or +1, i.e. attention to adjacent tokens).

55

Analysis on BERT
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, theRest Can Be Pruned (Voita et al., 2019)● There are different types of heads.
● Syntactic heads: comparing its attention weights to a predicted dependency
structure generated using CoreNLP.

https://corenlp.run/
56

Analysis on BERT
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, theRest Can Be Pruned (Voita et al., 2019)● There are different types of heads.
● Rare tokens: For all models, we find a head pointing to the least frequent
tokens in a sentence.

57

Analysis on BERTAre Sixteen Heads Really Better than One? (Michel, et al., 2019)● Make the surprising observation that even if models have been trained using
multiple heads, in practice, a large percentage of attention heads can beremoved at test time without significantly impacting performance.
● In fact, some layers can even be reduced to a single head.

58

Analysis on BERT
BERT Rediscovers the Classical NLP Pipeline (Tenney et al., 2019)●●

Quantify where linguistic
information is captured within the
network.
Find that the model represents
the steps of the traditional NLP
pipeline in an interpretable andlocalizable way, and that theregions responsible for each step
appear in the expectedsequence: POS tagging, parsing,
NER, semantic roles, thencoreference
59

BERT Code Demo
Hugging Face Transformers library
Hugging Face course
Code Demo● BERT for predicting masked tokens.
● BART for summarization● GPT-2 for text generation.

60

Can a LM ever truly understand natural language?
https://newsletter.ruder.io/issues/palm-dall-e-2-chinchilla-chain-of-thought-prompting-values-and-culture-in-nlp-845878?utm_campaign=Issue&utm_content=view_in_browser&utm_medium=email&utm_source=NLP+News
https://newsletter.ruder.io/issues/ml-and-nlp-starter-toolkit-low-resource-nlp-toolkit-can-a-lm-unders
tand-natural-language-the-next-generation-of-nlp-benchmarks-254211?utm_campaign=NLP%20N
ews&utm_medium=email&utm_source=Revue%20newsletter
https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00412/107385/Provable-Limitations-of-Acquirin
g-Meaning-from?utm_campaign=NLP%20News&utm_medium=email&utm_source=Revue%20new
sletter
https://arxiv.org/abs/2004.10151

61