In gradient descent we start at some point on the error function defined over the weights, and attempt to move to the global minimum of the function. In the simplified function of Fig 1a the situation is simple. Any step in a downward direction will take us closer to the global minimum. For real problems, however, error surfaces are typically complex, and may more resemble the situation shown in Fig 1b. Here there are numerous local minima , and the ball is shown trapped in one such minimum.
Progress here is only possible by climbing higher before descending to the global minimum.
The noise in the stochastic error surface is likely to bounce the network out of local minima as long as they are not too severe. Momentum Another technique that can help the network out of local minima is the use of a momentum term. This is particularly useful when the network is not well-conditioned. In such cases the error surface has substantially different curvature along different directions, leading to the formation of long narrow valleys. 2b shows how the addition of momentum helps to speed up convergence to the minimum by damping these oscillations.