Nagoya University, Fall 2022

Lecture 7

Naive Bayes II & Gaussian discriminant analysis

https://www.henrikbachmann.com/mml2022.html

Recall

Generative vs Discriminative learning algorithm
Logistic regression

x: feature (e.g. hours of studying)
y: label (e.g. passing or failing exam)

Want to find a hypothesis which describes P(y|x)
Learning P(y|x) is an example of a discriminative learning algorithm.

Generative learning algorithm: Learn P(x|y) and P(y) and then use
Bayes rule
together with the Naive Bayes assumption thatand the law of total probability

.

RecallEmail spam filter
Let's assume we want to create an email spam filter byusing supervised learning.
Labels: No Spam & SpamFeatures: EmailEmails
Training set

Email text

No spam / Spam

1 Do math today

02 Buy1

3 Buy book0

4 Today do math drugs

1

5 Buy drugs book today

1

According to this training set: Is the email “Buy book today” spam?

.

RecallEmail spam filter: From emails to vectors
Training set

Email text

No spam / Spam1Do math today

02Buy

1

3Buy book0

4Today do math drugs

15

Buy drugs book today1Create a dictionary and assign to an email a vector
1book

2buy

3do4drugs

5math

6today“Buy book today”

We will ignore the order and the number of appearances of words.

RecallEmail spam filter
Training set

Email text

No spam / Spam

1 Do math today

0

2 Buy

1

3 Buy book0

4 Today do math drugs

1

5 Buy drugs book today

1

The probabilities according to our training set:.

RecallEmail spam filter“Buy book today”

.

RecallNaive Bayes assumption
Naive Bayes assumption:
The features are “conditionally independent” given the label.

conditionally independent
A and B are conditionally independent given C if and onlyif, given knowledge that C occurs, knowledge of whether
A occurs provides no information on the likelihood of Boccurring, and knowledge of whether B occurs provides
no information on the likelihood of A occurring.

.

RecallConditionally independent: Example
A: Ability to do math of a person
B: Foot size of a person
A and B are not independent, since if I tell you someone's
foot size, it hints at their age, which in turn hints at theirability to do math.

C: Age of a person
A and B are conditionally independent given C, since if tell you
someone's age (C), then the ability of doing math (A) will notchange whether you know the foot size (B) or not, i.e.

P(A | C) = P(A | B,C).

RecallNaive Bayes classifier: Training
Indicator function

.

RecallNaive Bayes classifier: Judging a new email
Training: Determine the values.

Naive Bayes classifier: Laplace smoothing
We should never assume an event to have probability 0 or 1.
For this we will use Laplace smoothing and change our parameters as follows:

Interpretation: Add 4 imaginary emails••••

A spam email which contains every word.
A spam email which does not contain any word.
A non-spam email which contains every word.
A non-spam email which does not contain any word.

.

Naive Bayes classifier: Laplace smoothinghttps://colab.research.google.com/drive/1XU8NcI
Wbf1io_dDbMKLKqnKuy9MmNqu2?usp=sharing.

Normal/Gaussian distribution

A lot of things follow a normal distribution, e.g. height of people, exam results, etc.

.

Multivariate normal distribution”x follows a multivariate
normal distribution”

.

Multivariate normal distribution.

Gaussian discriminant analysis (GDA) model

Goal: Given a training set, find the best possible parameters

.

Gaussian discriminant analysis (GDA) modeld=2

.

Gaussian discriminant analysis (GDA) model