Geometric Intuition for Training Neural Networks
Channel: Seattle Applied Deep Learning
Date: 2019-11-25

Leo Dirac (@leopd) gives a geometric intuition for what happens when you train a deep learning neural network.  Starting with a physics analogy for how SGD works, and describing the shape of neural network loss surfaces.

This talk was recorded live on 12 Nov 2019 as part of the Seattle Applied Deep Learning (sea-adl.org) series.

References from the talk:
Loss Surfaces of Multilayer networks https://arxiv.org/pdf/1412.0233.pdf
Sharp minima papers:
   -Modern take https://arxiv.org/abs/1609.04836
   -Hochreiter, Schmidhuber 1997 http://www.bioinf.jku.at/publications/older/3304.pdf
SGD converges to limit cycles: https://arxiv.org/pdf/1710.11029.pdf
Entropy-SGD: https://arxiv.org/abs/1611.01838
Parle: https://arxiv.org/abs/1707.00424
FGE: https://arxiv.org/abs/1802.10026
SWA: https://arxiv.org/pdf/1803.05407.pdf
SWA implementation in pytorch: https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/