Nagoya University, Fall 2023

Lecture 13
PCA II & Autoencoder
This week Tutorial: Thursday 25th Jan. 6th period
This is also the deadline of Homework 4 and the semester project

https://www.henrikbachmann.com/mml2023.html

Semester project
Objective: Choose a project topic related to Nagoya or Japan more broadly thatcan be addressed using machine learning algorithms. Your task is to develop amachine learning model to solve a specific problem or provide insights into an
aspect of life, business, environment, culture, etc., in Nagoya/Japan.
Group Size: 1-3 members

A variation of Homework 2,3 or 4 is also ok!

Code: Preferably a Google Colab notebook. Exceptions are possible; pleaseprovide full documentation for any different technology or package used. If youplan not to submit a Google Colab, please contact us in advance.
Documentation: 5-10 slides as if you were going to present the project.
Your slides could cover for example:• Problem Statement• Data Collection• Data Exploration and Visualization• Model Building and Evaluation• Conclusion

Unsupervised learning: Dimensionality reduction
Example: Digit recognition (MNIST Dataset)
Pictures of size 28 x 28 = 784 pixels

Neural network with input layer size X

784 Datapoints ordered by “importance”.
Choose the first X of them.

Better approach: Find a new representation of thepicture into principal components.

Unsupervised learning: Dimensionality reduction
The first three principal components (PC) of the MNIST dataset

…
PC1

PC2

PC3

Principal Component Analysis (PCA)
Principal Component Analysis (PCA) is a method that reduces thedimensionality of data by transforming it into principal components, each
representing unique variance, while retaining the most significantinformation.

“Finds a better coordinate system for given data, suchthat the axes are ordered by importance”

Recall: Orthonormal bases

HARTEHENx̅

Recall: Orthonormal basesix AFFE

PCA – Tebasaki Example

PCA – Tebasaki Example

WGoal:IT• Represent these datapoints by 2 (PC1,PC2) values insteadof 3 (W,I,T)• Make it possible to get back (a good approximation of)(W,I,T) from (PC1,PC2)

PCA/Tebasaki Example: Step 1 - Normalize
We first normalize the data and give them a mean 0 and standard deviation of 1

Step 1mean = 0
std = 1

PCA/Tebasaki Example: Step 2 – Covariance matrix
Last lecture: The directions of the biggest variations are given (ordered bythe eigenvalue) by the eigenvectors of the covariance matrix.

Step 1mean = 0
std = 1X=PCA/Tebasaki Example: Step 2 – EigenvectorseLinear algebra fact (Spectral theorem): One can always find an ONBof eigenvectors for the covariance matrix (since it is symmetric).

PCA/Tebasaki Example: Step 3 – Getting PCs
There are 3 principal components (eigenvectors). Taking all of themwould not reduce the number of values we use to describe our point.
We could try to just use the first two.
Consider the projection (given by takingthe dot product) onto the plane spannedby the first two eigenvectors.

EL
3X
E feiert

Katie
OxPCA/Tebasaki Example: Step 3 – Plotting PC1,2E

HÄ
R

L

E
2 dimensional representation of our data

Now Python examples!

PCA – different interpretation
What we did: Have data in 3 dimensions, reduce it to 2 dimensions such that (if we go backto 3) we do not lose a lot of information.

WW’
PC1

II’
PC1z

TT’

PCA – different interpretation
What we did: Have data in 3 dimensions, reduce it to 2 dimensions such that (if we go backto 3) we do not lose a lot of information.
WW’
PC1

II’
PC1

TT’

This can be seen as a neural network without bias andwith the identity function as an activation function.
This is the basic idea of an autoencoder.

Autoencoder

Autoencoder Application: Deep fakes