CSE 587: Deep Learning for Natural
Language Processing
Lecture 2. Text Classification and Language Modeling

Rui Zhang
Spring 2023

1

Outline - Key Concepts
NLP● Text Classification● Language Modeling● Word EmbeddingML● Discriminative Model vs Generative Model● Objective Function● Gradient Descent● Evaluation● Statistical Testing● Feed-forward Neural Networks and Recurrent Neural Networks
2

Text Classification
Input X
Text
Text
Text

Output Y

Task

Label
Linguistic Structure
Text

Text Classification (e.g., Sentiment Analysis)
Structured Prediction (e.g., Part-of-Speech Tagging)
Text Generation (e.g., Translation, Summarization)

Classify sentences/documents into different classes.
● e.g., Sentiment Analysis: Positive, Neutral, Negative3

Discriminative Model
Input X
Text
Text
Text

Output Y

Task

Label
Linguistic Structure
Text

Text Classification (e.g., Sentiment Analysis)
Structured Prediction (e.g., Part-of-Speech Tagging)
Text Generation (e.g., Translation, Summarization)

Classify sentences/documents into different classes.
● e.g., Sentiment Analysis: Positive, Neutral, Negative
Discriminative Model: Calculate the conditional probability distribution of classlabels Y given the input data X.

4

From Prediction Score to Probability5Sigmoid for Binary Classification

6

Softmax for Multi-class Classification
Softmax extends the idea of sigmoid into a multi-class classification.
It converts scores to a probability distribution of class labels.

The predicted probability for the j'th class given a sample vector x and aweighting vector w is7

Softmax Example

https://deepai.org/machine-learning-glossary-and-terms/softmax-layer

8Softmax in Neural Networks

https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax

9

Discriminative Model
The discriminative model is parameterized by10

Discriminative Model Objective Function
The discriminative model is parameterized by

We often use negative log likelihood over training data as our objective function
or loss function. It is a function of

11

Discriminative Model Objective Function
The discriminative model is parameterized by

We often use negative log likelihood over training data as our objective function
or loss function. It is a function of

We then optimize the parameters to minimize the loss. Better model has lower loss
12

Optimize Objective Function by Gradient DescentCalculate the gradient of the loss function with respect to the parameter

Updateby moving a small step in the gradient direction to decrease the loss

is the learning rate
13

Gradient Descent14

Evaluation
Accuracy

https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers

15

Evaluation
Precision

RecallF1 score: the harmonic mean of precision and recall●harmonic mean < geometric mean < arithmetic meanhttps://en.wikipedia.org/wiki/Precision_and_recall

16

Precision-Recall Curve

https://medium.com/@douglaspsteen/precision-recall-curves-d32e5b290248

17

ROC Curve
Recall is also called True Positive Rate
● True Positive / (True Positive + False Negative)
We can also define False Positive Rate
● False Positive / (False Positive + True Negative)
Be default, we can use 0.5 as threshold, but we canuse other threshold as well.
As we change the threshold, both TPR and FPRchange.
ROC curve: Receiver Operating Characteristic

18

ROC Curve
Recall is also called True Positive Rate
● True Positive / (True Positive + False Negative)
We can also define False Positive Rate
● False Positive / (False Positive + True Negative)

https://towardsdatascience.com/understanding-auc-roc-curve-68b2303
cc9c5

19

AUC
AUC: Area under Curve● AUC = 1: The perfect classifier● AUC = 0.5: The random classifier● AUC = 0: The worst classifier

20

Evaluation for Ranking
Classification: order of predictions doesn't matter
Ranking: order of predictions does matter● Search engines: Predict which documents
match a query on a search engine.

https://queirozf.com/entries/evaluation-metrics-for-ranking-problems-introduction-and-examples

21

Evaluation for Ranking
Average Precision● It is the area under the precision-recall curve.
● It is equivalent to compute the following

where rel(k) = 1 if the item at rank k is relevant, 0otherwise.
AveP = (1/1 + 2/3 + 3/4 + 4/6) / 4

22

Evaluation for Ranking
Mean Average Precision● average of AP over all examples in a test set.
There are many metrics for ranking.
● DCG/NDCG: the document relevance is a real number, not simple 0 or 1.

23

Evaluation for Natural Language Generation(We will talk about this in more detail when we have lectures on NLG andSummarization.)
Automatic Evaluation● Machine Translation: BLEU
● Summarization: ROUGE, METEOR● Embedding-based Metric: MoverScore, BERTScore
Human Evaluation
Is Automatic Metric really reliable? Correlation Analysis
24

Statistical Testing
We have two models with similar accuracies. How can we tell whether thedifferences are due to consistent trends that hold on other datasets?

We need perform Statistical (significance) testing!
See The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language
Processing (Dror et al., ACL 2018) for a complete overview.
Slides Credit: Graham Neubig

25Significance Testing: Basic Idea
Given a quantity, we test certain values of uncertainty with respect to the quantity,e.g.
●

p-value: what is the probability that a difference with another quantity is bychance (lower = more likelihood of a significant difference).

●

confidence interval: what is the range under which we could expect another
trial to fall?

Slides Credit: Graham Neubig

26

Unpaired vs. Paired Tests
Unpaired Test: Compare means of a quantity on two unrelated groups● Example: test significance of difference of accuracies of a model on twodatasets
Paired Test: Compare means of a quantity on one dataset under two conditions● Example: test significance of difference of accuracies of two models on one
datasetWe are most commonly interested in Paired Test!

Slides Credit: Graham Neubig

27

Bootstrap Tests
A method that can measure p-values, confidence intervals, etc. by re-sampling data.
Sample many (e.g. 10,000) subsets from your dev/test set with replacement.
Measure accuracies on these many subsets.
Easy to implement, applicable to any evaluation measure, but somewhat biased on smalldatasets.

Slides Credit: Graham Neubig

28

Discriminative Model vs Generative Model
Discriminative Model: Calculate the conditional probability distribution of classlabels Y given the input data X.

Generative Model: Directly Model

29

Language Modeling
Language Modeling: Calculate the probability of a sentence.
Sentence X of L words
Decompose the joint probability by chain rule30

Language Modeling
Language Modeling: Calculate the probability of a sentence.
Sentence X of L words
Decompose the joint probability by chain rule●
●We often deal with log probabilities because of numerical stabilities and other benefits.
Different ways for calculating the probability of next word give different language
modeling approaches.

31

Unigram Language Models
Assumption:

Count-based Maximum-likelihood estimation (MLE):

32

Bigram Language Models
Assumption:

Count-based Maximum-likelihood estimation (MLE):

33

Higher-order n-gram Models
Unigram (1-gram), Bigram (2-gram)
Trigram (3-gram)34

Feed-Forward Neural Network Language ModelsAssumption: The distribution of next word can be parameterized by feed neuralnetworks.

Feed Forward Neural Networks

Word Embedding

Word Embedding

A Neural probabilistic language model. Bengio et al., 2003

Word Embedding35

A fixed-window neural language model

https://people.cs.umass.edu/~miyyer/cs685/slides/02-neural-lms.pdf

36

Recurrent Neural Network Language Models
Assumption: The distribution of next word can be parameterized by recurrent
neural networks.

RNNRNN

RNN

Word Embedding

Word Embedding

Word Embedding

Recurrent neural network based language model. Mikolov et al., 201037

https://people.cs.umass.edu/~miyyer/cs685/slides/02-neural-lms.pdf

38Evaluation of Language Modeling
We train our language model on training set. How good is our language model?
A good language model should assign a high probability to text in test set!
Log-Likelihood:39

Evaluation of Language Modeling
We train our language model on training set. How good is our language model?
A good language model should assign a high probability to text in test set!
Log-Likelihood:

Per-word Log Likelihood:

Per-word Cross Entropy

40

Evaluation of Language Modeling - Perplexity
Perplexity is the inverse probability of the test set.
Lower perplexity, higher probability, better model.

41

Evaluation of Language Modeling - Perplexity
Perplexity is the inverse probability of the test set.
Lower perplexity, higher probability, better model.

Perplexity is the exponentiated token-level negative log-likelihood.
e.g., for bigram LM:

42

Usage of LMs
1. Score given sentences

2. Generate new sentences
while didn’t choose end-of-sentence symbol:
calculate the probability distribution of next word
sample a new word from the probability distribution of next word
3. Testbed for a neural networks

43

Usage of LMs

https://devopedia.org/language-modelling

44

Usage of LMs

Generate Text from LM trained on Shakespeare

45