Gradient descent is a first-order iterative optimization algorithm for finding the minimum of a function. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or approximate gradient) of the function at the current point. If, instead, one takes steps proportional to the positive of the gradient, one approaches a local maximum of that function; the procedure is then known as gradient ascent. The steps to use gradient descent algorithm is as follows: Initialize random weight and bias Pass an input through the network and get values from the output layer Calculate error between the actual value and the predicted value Go to each neurons which contributes to the error and change its respective values to reduce the error Reiterate until we find the best weights of the network Below is an illustrative diagram of gradient descent on a series of level sets.