Nagoya University, Fall 2023

Lecture 7

Neural Networks IV:

Finishing Backpropagation & Convolutional Neural Networks

This week Tutorial: Wednesday 22nd Nov. 5th period
https://www.henrikbachmann.com/mml2023.html

Semester project
Objective: Choose a project topic related to Nagoya or Japan more broadly thatcan be addressed using machine learning algorithms. Your task is to develop amachine learning model to solve a specific problem or provide insights into an
aspect of life, business, environment, culture, etc., in Nagoya/Japan.
Group Size: 1-3 members
Code: Preferably a Google Colab notebook. Exceptions are possible; pleaseprovide full documentation for any different technology or package used. If you
plan not to submit a Google Colab, please contact us in advance.
Documentation: 5-10 slides as if you were going to present the project.
Your slides should cover (for example):• Problem Statement• Data Collection• Data Exploration and Visualization• Model Building and Evaluation• Conclusion

Semester project• The notebook should contain outputs (we might not run/train)• Kaggle.com for possible datasets• You do not need to cure cancer• A “bad result” is also a result• Be able to answer questions about your project in person• We are not strict about the “Japan & Nagoya” connection

Neural Network: forward pass…

Backpropagation

For gradient descent, we need to calculate the gradient of J.

Backpropagation: A simple example in one dimensions

N R

M

TERp

ERWARB
ZE
WAR
fürCAM

ÜBER

EEN

W

TIO

I

warten.fi

K2

DER

EINEN YT

Elam Y1

FEH

Want DTIEEINHYT

JaffeµwannaE
E

Eec

KETTE

W
STIEHT

Recallchainrul y

EETTE.EEglxl EFIÖEIRETTEFEETFwsinFEIBackpropagation: General idea

Ja Ja

Want…

WII

IIIIE

paix

mitsie8misuseTEEEzGeneral chain rule

9JJE

109J gFESTOp9m
5ER

General Backpropagation

WantSEI
5JE
54

Ei

JA

ja

14cm

EkinDscFüiiz

WüEisX

SEgWang

Ö zig

ff

Wr

Component wise

multiplication

Backpropagation

See Lecture 6 notebook

Neural Networks so far
The fully connected Neural Networks we discussed so far
come with some problems:
1.High Parameter Count: Excessive parameters lead to computational and memory
inefficiency.
2.Overfitting Risk: Prone to overfitting due to a large number of parameters.
3.Spatial Inefficiency: Inability to efficiently process spatial data and structure in images.
4.No Translation Invariance: Lacks inherent ability to recognize shifted or translated
features.
5. Unsuitable for (large) Images: Increasing image size leads to a disproportionate increasein parameters.

Convolutional Neural Networks (according to ChatGPT)

Convolutional Neural Networks (more Japanese)What is convolution?

IPicture

FilterKernel

2 YFeaturkap

Convolutional Neural Networks

Types of layers
1.Dense (Fully Connected) Layer: Every neuron in this layer isconnected to all neurons in the previous layer, commonly used forclassification or regression tasks. (the only thing we considered so far)
2.Convolutional Layer: Applies a set of learnable filters to extract
spatial features from data like images.
3.Pooling Layer: Reduces the spatial dimensions (width, height) of
the input volume, commonly used for downsampling. Examples
include Max Pooling and Average Pooling. (e.g. 2x2-MaxPooling)
4.Dropout Layer: Randomly sets a fraction of the input units to 0
during training, helping prevent overfitting.

CNN for MNIST

https://www.kaggle.com/code/amyjang/tensorflow-mnist-cnn-tutorial