Nagoya University, Fall 2023

Lecture 12
Principal Component Analysis (PCA)
This week Tutorial: Friday 19th Dec. 6th period
https://www.henrikbachmann.com/mml2023.html

Semester project
Objective: Choose a project topic related to Nagoya or Japan more broadly thatcan be addressed using machine learning algorithms. Your task is to develop amachine learning model to solve a specific problem or provide insights into an
aspect of life, business, environment, culture, etc., in Nagoya/Japan.
Group Size: 1-3 members
Code: Preferably a Google Colab notebook. Exceptions are possible; pleaseprovide full documentation for any different technology or package used. If you
plan not to submit a Google Colab, please contact us in advance.
Documentation: 5-10 slides as if you were going to present the project.
Your slides should cover (for example):• The slides are the documentation and• Problem Statement
should explain what you did.
• Data Collection• Do not just include examples and• Data Exploration and Visualization
diagrams.
• Model Building and Evaluation• It should be possible to understand• Conclusion
your project by just reading the slides.

Overview

Unsupervised learning• Clustering (Lecture 11)• Anomaly detection• Signal separation•…

https://en.wikipedia.org/wiki/Signal_separation
https://en.wikipedia.org/wiki/Cocktail_party_effectLecture 11: k-means clustering

Result after 20 steps

Unsupervised learning: Dimensionality reduction
Example: Digit recognition (MNIST Dataset)
Pictures of size 28 x 28 = 784 pixels
Neural network with input layer size 784

Question: How can we reduce the size of the input layer
without losing a lot of information?

Unsupervised learning: Dimensionality reduction
Example: Digit recognition (MNIST Dataset)
Pictures of size 28 x 28 = 784 pixels

Neural network with input layer size 64

Pictures of size 8 x 8 = 64 pixels

Naive approach: Scale the image down or ignore pixels

Unsupervised learning: Dimensionality reduction
Example: Digit recognition (MNIST Dataset)
Pictures of size 28 x 28 = 784 pixels

43

fit

Neural network with input layer size X

I784 Datapoints ordered by “importance”.
Choose the first X of them.

Better approach: Find a new representation of thepicture into principal components.

Unsupervised learning: Dimensionality reduction
The first three principal components (PC) of the MNIST dataset

…
PC1

PC2

PC3

Principal Component Analysis (PCA)
Principal Component Analysis (PCA) is a method that reduces thedimensionality of data by transforming it into principal components, each
representing unique variance, while retaining the most significantinformation.

“Finds a better coordinate system for given data, suchthat the axes are ordered by importance”

PCIIHefenPa1oPCA – Variance

Given numbers XExinCR

averageµvariance

Var XXi µVartanVarte

Xi

PCA – Orthogonal projection
U

Given UER

Tuorthogonal projection

Pakt Ian

If Hull 1
Pulxtun1

if Hall 1u

PCA – First principal component
How to find the first “principal component”

d 2
1

Given

513413

Steph

RdrienScale sithO

mean

i

Standard deviation

1

PCA How to find the first “principal component”
after scaling

Want
we

weHull 1

Rd such thatmaximizeIx nldaÜh

TÄTEw̅

Euu

dT.DE

ixiTuX

Yncovariancematrix

In example

Ecovarianceof room site

AE Rprice0eigenvalue X eigenvector

AEAl01 1

A 91

18219

SummaryForsymmetric M we wanttomaximize

ut Mu

Theoreme For any sym.ME RIII

the term
is maximal

ifud

Rayleighquotientis aneigenvector

Of M for the largest eigenvalue

Prooftheorem
thespectral
By

QDQXH.PH

M
Assume Hull 1

DIM U

UTQD QUa

QUID QU

Qy

QUYTY

YTDY
11411
a

1y.FI

yTDy

2XiY4

11maximal forthen it isaQy
firstcolumnof Qeigenvector fort