Contrastive Learning for Natural
Language Processing
Rui Zhang, Penn State University
https://ryanzhumich.github.io/
August 4, 2022 @ Amazon

Contrastive Learning
Learning embeddings such that similar data sample pairs are close while
dissimilar sample pairs stay far apart (Chopra et al., 2005)

Figure from Khosla et al., 2020

Contrastive Learning in Computer Vision
SimCLR (Chen et al., 2020)

Most Successful Example of Contrastive Learning for NLP
word2vec (Mikolov et al., 2013) for word embeddings

word2vec’s skip-gram model. Figure from Chris McCormickAgenda●●Part 1: Foundations of Contrastive Learning
Part 2: Contrastive Learning for NLP: A Case Study in Named Entity Recognition

Part 1.
Foundations of Contrastive Learning

Two Elements of Contrastive Learning
Contrastive Learning = Contrastive Data Creation + Contrastive Objective Optimization

Part 1.1
Contrastive Learning Objectives

Noise Contrastive Estimation (NCE)
Use Logistic Regression with cross-entropy loss to differentiate positive samples(i.e., target distribution) and negative samples (i.e., noise distribution).
Logit function of a sample from the target distribution
Probability a sample from the target distribution

Noise-contrastive estimation: A new estimation principle for unnormalized statistical models (Gutmann and Hyvärinen, 2010)

InfoNCE
Use softmax loss to differentiate a positive sample from a set of noise examples.

Context Vector, e.g., anchor point
N samples with 1 positive sample and N-1 negative samples

1 positive sample
1 positive sample + N-1 negative samples

Representation Learning with Contrastive Predictive Coding (van den Oord et al., 2018)

Normalized Temperature-scaled Cross-Entropy (NT-Xent)

Cosine Similarity
Normalized Embeddings
Temperature controls the relative importance of the distances between point pairs●●At low temperatures, the loss is dominated by the small distances.
At high temperatures, the loss is dominated by the large distances.

A Simple Framework for Contrastive Learning of Visual Representations. (Chen et al., 2020)

Contrastive Lossminimizes the embedding distance when
they are from the same classmaximizes the embedding distance when they
are from the same class

Learning a Similarity Metric Discriminatively, with Application to Face Verification (Chopra et al., 2005)

Triplet Loss

We push the the distance between positive and anchor + margin to be smaller than thedistance between negative and anchor.
FaceNet: A Unified Embedding for Face Recognition and Clustering (Schroff et al., 2015)

N-pair Loss
Triplet Loss●●●

N-pair Loss

Extend to N-1 negative examples
Inner product similarity + softmax loss
Similar to multi-class classification

Improved Deep Metric Learning with Multi-class N-pair Loss Objective (Sohn, 2016)

Lifted Structured Loss
Lifted Structured Loss explicitly
takes into account all pairwiseedges within the batch.

Illustration for a training batch with six examples.
Red edges: similar examples.
Blue edges: dissimilar examples.

Deep Metric Learning via Lifted Structured Feature Embedding (Song et al., 2016)

Summary of Contrastive Learning Objectives

Part 1.2
Contrastive Data Sampling and Augmentation Strategies

Self-Supervised Contrastive Learning
Positive: Data Augmentation
Negative: Random, e.g., In-batch Negatives
The Biggest Advantage: No label is required!

Figure from (Khosla et al., 2020)

Four Challenges of Self-Supervised Contrastive Learning
1.
2.
3.
4.

Non-trivial Data Augmentation
Risk of “Sampling Bias” (i.e., False Negative)Hard Negative Mining
Large Batch Size

Figure from (Khosla et al., 2020)

Data Augmentation for Text
Text Space●●

Lexical Editing (token-level)
Back-Translation (sentence-level)

Embedding Space●●●DropoutCutoff
Mixup

Manual
EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. (Wei and Zhou, 2019)
Conditional BERT Contextual Augmentation (Wu et al., 2018)
Improving Neural Machine Translation Models With Monolingual Data (Sennrich et al., 2016)
CERT: Contrastive Self-supervised Learning for Language Understanding (Fang et al., 2020)
SimCSE: Simple Contrastive Learning of Sentence Embeddings. (Gao et al., 2021)
A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation. (Shen et al., 2020)
mixup: Beyond Empirical Risk Minimization. (Zhang et al., 2017)
NL-Augmenter A Framework for Task-Sensitive Natural Language Augmentation (Dhole et al., 2021)

Sampling Bias

Problem: Because we don’t know the label, we may accidentally create false negative by
sampling examples from the same class.

Debiased Contrastive Learning (Chuang et al., 2020)

Debiased Contrastive Learning
Key Idea: Assume a prior probability between positive and negative, then approximate thedistribution of negative examples to debias the loss.

Then samples N samples (may contain positive and negative) and M positive samples

Debiased Contrastive Learning (Chuang et al., 2020)

Hard Negative Mining

A: Anchor. P: Positive.

N: Negative

We want to AN is greater than AP, at least by the margin.
Hard Negative Mining: Find hard negatives

Figure from Kurowski et al., 2021

Hard Negative Mining by Importance Sampling

new sampling
probability

similarity

original sampling
probabilityKey Idea: If this negative sample is close to the anchor
sample, then we up-weight its probability of being
selected.
Figure from Kurowski et al., 2021
Contrastive Learning with Hard Negative Samples (Robinson et al., 2021)

Large Batch Size

SimCLR of ResNet-50 trained withdifferent batch sizes and epochs.

“We train with larger batch size (up to 32K) and longer (up to 3200 epochs).”— Chen et al., SimCLR“We use a very large minibatch size of 32,768.”— Radford et al., CLIP

Memory Bank to Reduce Computation
Memory Bank: Compute and store the representations in advances, instead ofcomputing embeddings for all examples in a batch.

Instance-level discrimination uses contrastive learning to maximally scatter the features of training
samples over the 128-dimensional unit sphere. Embeddings are stored in a Memory Bank.
Unsupervised Feature Learning via Non-Parametric Instance Discrimination. (Wu et al., 2018)

From Self-Supervised to Supervised Contrastive Learning

Supervised Contrastive Learning (Khosla, et al., 2020)

Supervised Contrastive Learning
Positive: Same Class
Negative: Different ClassPros● No Need for Data Augmentation● No Risk of “False Negative”● No Need for Large Batch SizeCons● Need Label
Sentence-BERT, SimCSE, DPR, CLIP
Supervised Contrastive Learning (Khosla, et al., 2020)

Part 2.
Contrastive Learning for NLP: A Case Study in Named
Entity Recognition

Contrastive Learning for NLP(Smith and Eisner, 2005): The first NLP paper introducing “contrastive estimation”
as an unsupervised training objective for log-linear models.

“neighborhood” N(xi) is a set of implicit negative
examples plus the example xi itself.

Why Talk about Contrastive Learning for NLP Today?
Number of papers with titles containing “contrastive learning” in recent NLP conferences

Why Talk about Contrastive Learning for NLP Today?
Number of papers with titles containing “contrastive learning” in recent NLP conferences

Contrastive Learning for NLP●word embeddings
○
○
○
○

●

various tasks.

Classification: Text Classification, Information Extraction
Reasoning: Commonsense Reasoning, Question Answering, Fact Verification
Generation: Summarization, Machine Translation, Text Generation
Multimodal Learning: Vision-and-Language

performance improvements○○○○

sentence representations

desired characteristics

Task-agnostic Sentence Representation
Data-efficient Learning in Zero-shot and Few-shot settings
Interpretability and Robustness
Faithful Text Generation

Contrastive Learning for NLP●word embeddings
○
○
○
○

●

various tasks.

Classification: Text Classification, Information Extraction
Reasoning: Commonsense Reasoning, Question Answering, Fact Verification
Generation: Summarization, Machine Translation, Text Generation
Multimodal Learning: Vision-and-Language

performance improvements○○○○

sentence representations

desired characteristics

Task-agnostic Sentence Representation
Data-efficient Learning in Zero-shot and Few-shot settings
Interpretability and Robustness
Faithful Text Generation

CONTaiNER: Few-shot Named Entity Recognition
Using Contrastive Learning
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J. Passonneau, Rui Zhang
ACL 2022

Named Entity Recognition (NER)
Barack Obama (born August 4, 1961) is an American attorney and politician who
served as the 44th President of the United States from January 20, 2009 toJanuary 20, 2017.
PersonDate

Location

Other (“O”)

Few-Shot Named Entity Recognition
Barack Obama (born August 4, 1961) is an American attorney and politician who
served as the 44th President of the United States from January 20, 2009 toJanuary 20, 2017.
PersonDateLocation

Other (“O”)

Traditionally, we have a training dataset with sufficient examples for each of thecategories (e.g., OntoNotes).
Entities from low resource domains (e.g., Medical) suffer from the scarcity of data.
Can we learn NER models on new domains / new categories with only a few
examples?

Main Idea: Contrastive Learning over Token Representations

Idea 2: Gaussian Token Embeddings●●Model embeddings as distributions instead of points
Word Representations via Gaussian Embedding (Vilnis and McCallum, ICLR 2015)○ The token embeddings follow some Gaussian Distribution (mean µ, diagonalcovariance σ).
BarackNelson

Contrastive Learning with Gaussian Embeddings

Idea 3: Few-shot Learning from Source to Target

Datasets

Few-NERD (Ding et al., 2021)

Generalization to Unseen Tags

Generalization to Unseen Domains

Results on Few-NERD
Few-NERD (Inter): train and test classes canshare coarse grained types

Few-NERD (Intra): Train and Test classes do
not share their coarse-grained types, whichmakes it more challenging.

Point Embeddings vs Gaussian Embeddings

Future Work
Few-shot Sequence Labelling●●How can we create few-shot learning models for structured predictions?
NER -> Joint entity and relation extraction, Semantic role labeling,
Coreference resolution, Semantic Parsing, ……

Contrastive Pretraining●●●

Is MLM the best pretraining strategy?
Contrastive Pretraining based on sentence embeddings is very slow.
How can we use contrastive learning on tokens to do pretraining?

CONTaiNER Codehttps://github.com/psunlpgroup/CONTaiNER

Full Version of Tutorial
https://contrastive-nlp-tutorial.github.io/Thanks! Any Questions?
Contact●●●

https://ryanzhumich.github.io/
rmz5227@psu.edu
https://github.com/psunlpgroup