Nagoya University, Fall 2023

Lecture 3

Polynomial regression & Logistic regression

https://www.henrikbachmann.com/mml2023.html

Lecture notes & Tutorial• You can now find the first version of the lecture noteson the homepage. (may contain typos)• Anyone interested in helping out with writing them(latex in overleaf) feel free to contact me.

OctoberThis week the Tutorial is Friday 6th period! 271h

1 Supervised learning: Notations

Recall• Learning algorithms make an Ansatz (educatedguess) for a hypothesis involving certainparameters.
• Learning: Find good parameters depending onthe trainings set.

1 Supervised learning – Linear Regression

RecallLearning Algorithm: Linear Regression

Goal: Minimize the cost function for a given trainings set.

Recall

1 Supervised learning – Solution 1: Gradient descent

Do severaltimes

Learning rate

Recall

1 Linear Regression: Solution 2 - Use linear algebra

NormalequationFd 2X

X

RE91,4999

Ä
D9 ER
EHEIE

Al

half00

Ifo

Solve04

9Atg

2

9IE H

KoF

1 Linear Regression – Not only for “linear data”
Linear regression can also be used for polynomialinterpolation and other types of functions.

hat_äh2 Binary Classification - Logistic Regression

2 Binary Classification – Passing exam
Binary Classification

Passing exam example

Features: Hours studied for the exam
Labels: Failed exam (y=0), Passed exam (y=1)

Erwantsigmoidfunction
Question: How to model the hypothesis?

2 Binary Classification – Logistic regression

Scottfingsatt

ftp.Sxt

2 Binary Classification – Logistic regression

Recall: Linear regression
Hypothesis:

Logistic regression
Hypothesis:

2 Binary Classification – Logistic regression
Passing exam example

Question: How to find the best weights?

2 Logistic regression – Probabilities
Notation: P(A|B) refers to the conditional probability thatevent A occurs, given that event B has occurred.

2 Logistic regression – Probabilities
Notation: P(A|B) refers to the conditional probability thatevent A occurs, given that event B has occurred.

2 Logistic regression – Maximum likelihood
Likelihood: “measures the goodness of fit of a statisticalmodel to a sample of data"
Likelihood of parameters = Product over all probabilities in the training set

2 Logistic regression – Maximum likelihood
Likelihood: “measures the goodness of fit of a statistical model to a sample of data"

Likelihood of parameters = Product over all probabilities in the training set

Goal: Given a training set, find the parameters with the maximal likelihood.

2 Logistic regression – Maximum likelihood - Example
µ191

Iget

PG x

PEoshota
095

nöSEE

0.93

go.gg
jesIIto2 Logistic regression – Maximum log likelihood• Often it is easier to maximize the logarithm of the likelihood.
• The logarithm is monotonically increasing.
• The logarithm turns products into sums.
HEGoal: Given a training set, find the parameters with the maximal log likelihood.

1 Binary Classification – Gradient ascent

We want to maximize the log likelihood by using gradient ascent.

Recall: Linear regression
Minimized the cost function J by gradient descent:2 Logistic regression – Gradient ascent

Gradient:2 Logistic regression – Gradient ascentnGradient:

ESYotaktRaiSEKTE

Ask ftp Texte

OdXdejiE2 Logistic regression – Gradient ascent2 Logistic regression – Gradient ascent2 Logistic regressionIn the colab notebook of Lecture 3 you cansee the gradient ascent for the passing exam
example.