Nagoya University, Fall 2022

Lecture 3

Logistic regression & Maximum likelihood

https://www.henrikbachmann.com/mml2022.html

Lecture notes• You can now find the first version of the lecture noteson the homepage. (may contain typos)• We try to update it weekly.
• Anyone interested in helping out with writing them(latex in overleaf) feel free to contact me.

1 Supervised learning: Notations

Recall• Learning algorithms make an Ansatz (educated guess)for a hypothesis involving certain parameters.
• Learning: Find good parameters depending on thetrainings set.

1 Supervised learning – Linear Regression

RecallLearning Algorithm: Linear Regression

Goal: Minimize the cost function for a given trainings set.

Recall

1 Supervised learning – Solution 1: Gradient descent

Do severaltimes

Learning rate

Recall

1 Linear Regression: Solution 2 - Use linear algebra

Normalequation1 Linear Regression: Homework 2

1 Linear Regression – Not only for “linear data”
Linear regression can also be used for polynomialinterpolation and other types of functions.

2 Binary Classification - Logistic Regression

2 Binary Classification – Passing exam
Binary ClassificationPassing exam example

Features: Hours studied for the exam
Labels: Failed exam (y=0), Passed exam (y=1)

Question: How to model the hypothesis?

2 Binary Classification – Logistic regression

2 Binary Classification – Logistic regression

Recall: Linear regression
Hypothesis:

Logistic regression
Hypothesis:

2 Binary Classification – Logistic regression
Passing exam example

Question: How to find the best weights?

2 Logistic regression – Probabilities
Notation: P(A|B) refers to the conditional probability thatevent A occurs, given that event B has occurred.

2 Logistic regression – Probabilities
Notation: P(A|B) refers to the conditional probability thatevent A occurs, given that event B has occurred.

2 Logistic regression – Maximum likelihood
Likelihood: “measures the goodness of fit of a statisticalmodel to a sample of data"
Likelihood of parameters = Product over all probabilities in the training set

2 Logistic regression – Maximum likelihood
Likelihood: “measures the goodness of fit of a statistical model to a sample of data"

Likelihood of parameters = Product over all probabilities in the training set

Goal: Given a training set, find the parameters with the maximal likelihood.

2 Logistic regression – Maximum likelihood - Example

2 Logistic regression – Maximum log likelihood• Often it is easier to maximize the logarithm of the likelihood.
• The logarithm is monotonically increasing.
• The logarithm turns products into sums.

Goal: Given a training set, find the parameters with the maximal log likelihood.

1 Binary Classification – Gradient ascent

We want to maximize the log likelihood by using gradient ascent.

Recall: Linear regression
Minimized the cost function J by gradient descent:

Next week:• Calculate the gradient of the log likelihood• Use gradient ascent to find the best weights

2 Logistic regression – Gradient ascent

Gradient:2 Logistic regression – Gradient ascent

Gradient:2 Logistic regression – Gradient ascent2 Logistic regression – Gradient ascent2 Logistic regression

In the colab notebook of Lecture 3 you can
see the gradient ascent for the passing exam
example.