Basic layers in deep learning韩坤洋A simple MLP• Set input• Forward, get output• Calculate loss
• Calculate gradient• Update network

A simple MLP• Set input• Forward, get output• Calculate loss
• Calculate gradient• Update network

A simple MLP• Set input• Forward, get output• Calculate loss
• Calculate gradient• Update network

Activation• Non-linear function• Why need it?

Activation• Non-linear function• Why need it?

Activation• Non-linear function• Why need it?

Activation• Non-linear function• Why need it?

Activation• Non-linear function• Why need it?

Activation

• sigmoid• tanh• ReLU• leakyReLU• eLU• softmax

Convolution layer• 1 dimensional conv, Time series data
• 2d conv, image• 3d conv, video

0 + 1 + 6 + 12 = 19

Convolution layer• Kernel size

Convolution layer• Input channel• Output channel

Convolution layer• Stride• PaddingConvolution layer• Dilation

Transposed Convolution layer

Upsample• Mode• Nearest• Bilinear• Align cornerUpsample• Mode• Nearest• Bilinear• Align corner

Pooling• Type: max pooling, avg pooling• Kernel size
• StrideNormalization• Make sure each sample have same distribution

Loss function• Regression• Mean Square Error• Mean Absolute Error• Huber loss
• Log-Cosh• Classification• Cross entropy• Hinge loss
• Exponential loss

Loss function• Regression• Mean Square Error• Mean Absolute Error• Huber loss
• Log-Cosh• Classification• Cross entropy• Hinge loss
• Exponential loss

Loss function• Regression• Mean Square Error• Mean Absolute Error• Huber loss
• Log-Cosh• Classification• Cross entropy• Hinge loss
• Exponential loss

Loss function• Regression• Mean Square Error• Mean Absolute Error• Huber loss
• Log-Cosh• Classification• Cross entropy• Hinge loss
• Exponential loss

Loss function• Regression• Mean Square Error• Mean Absolute Error• Huber loss
• Log-Cosh• Classification• Cross entropy• Hinge loss
• Exponential loss

Loss function
KL Divergence• Regression• Mean Square Error• Mean Absolute Error• Huber loss
• Log-Cosh• Classification• Cross entropy• Hinge loss
• Exponential loss

A measure of how one probability distribution isdifferent from a second.

Loss function• Regression• Mean Square Error• Mean Absolute Error• Huber loss
• Log-Cosh• Classification• Cross entropy• Hinge loss
• Exponential loss

Primarily in SVM
It is intended for use with binary classification
where the target values are in the set {-1, 1}.

yf(x)-1 or 1[-1, 1]

Loss function• Regression• Mean Square Error• Mean Absolute Error• Huber loss
• Log-Cosh• Classification• Cross entropy• Hinge loss
• Exponential lossOptimizer• Learning rate

Optimizer• Learning rate

Optimizer• Gradient Descent (GD)
• Batch Gradient Descent （BGD）• Stochastic Gradient Descent (SGD)• Mini-Batch Gradient Descent （MBGD）• Momentum• Adagrad• Adadelta• RMSprop• AdamOptimizer• Gradient Descent (GD)
• BGD
• SGD
• MBGD• Momentum• Adagrad• Adadelta• RMSprop• AdamOptimizer
LearningRate = LearningRate * 1/(1 + decay * epoch)• Gradient Descent (GD)
• BGD
• SGD
• MBGD• Momentum• Adagrad• Adadelta• RMSprop• AdamAdagrad

Optimizer• Gradient Descent (GD)
• BGD
• SGD
• MBGD

1

2• Momentum• Adagrad• Adadelta• RMSprop3• Adam Adaptive Moment Estimation