In training deep networks, it is usually helpful to anneal the learning rate over time. Good intuition to have in mind is that with a high learning rate, the system contains too much kinetic energy and the parameter vector bounces around chaotically, unable to settle down into deeper, but narrower parts of the loss function. Knowing when to decay the learning rate can be tricky: Decay it slowly and you’ll be wasting computation bouncing around chaotically with little improvement for a long time. But decay it too aggressively and the system will cool too quickly, unable to reach the best position it can. There are three common types of implementing the learning rate decay. Step decay - Reduce the learning rate by some factor every few epochs. Typical values might be reducing the learning rate by a half every 5 epochs, or by 0.1 every 20      epochs. These numbers depend heavily on the type of problem and the model. Exponential decay – has the mathematical form alpha = alpha0 * exp(-kt); where alpha0, k are hyperparameters, and t is the iteration number 1/t decay – has the mathematical form alpha = alpha0 / (1 + kt) , where alpha0, k are hyperparameters, and t is the iteration number