Week 2 – Lecture: Stochastic gradient descent and backpropagation
Channel: Alfredo Canziani
Date: 02/03/2020

Course website: http://bit.ly/DLSP20-web
Playlist: http://bit.ly/pDL-YouTube
Speaker: Yann LeCun
Week 2: http://bit.ly/DLSP20-02

0:00:00 – Week 2 – Lecture

LECTURE Part A: http://bit.ly/DLSP20-02-1
We start by understanding what parameterized models are and then discuss what a loss function is. We then look at gradient-based methods and how it's used in the backpropagation algorithm in a traditional neural network. We conclude this section by learning how to implement a neural network in PyTorch followed by a discussion on a more generalized form of backpropagation.
0:00:29 – Gradient Descent Optimization Algorithm
0:17:16 – Advantages of SGD, Backpropagation for Traditional Neural Net
0:38:08 – PyTorch implementation of Neural Network and a Generalized Backprop Algorithm

LECTURE Part B: http://bit.ly/DLSP20-02-2
We begin with a concrete example of backpropagation and discuss the dimensions of Jacobian matrices. We then look at various basic neural net modules and compute their gradients, followed by a brief discussion on softmax and logsoftmax. The other topic of discussion in this part is Practical Tricks for Backpropagation.
0:49:49 – Basic Modules - LogSoftMax
1:05:53 – Practical Tricks for Backpropagation
1:21:31 – Computing gradients for NN modules and Practical tricks for Back Propagation