page_title: Backpropagation
section_title: Finding the derivative of the error
hierarchical_section_title: Backpropagation / Derivation / Finding the derivative of the error
caption_attribution_description: English: Diagram of an artificial neuron. Français&#160;: Schéma d'un neurone artificiel.
context_page_description: In machine learning, backpropagation is a widely used algorithm in training feedforward neural networks for supervised learning. Generalizations of backpropagation exist for other artificial neural networks, and for functions generally – a class of algorithms referred to generically as "backpropagation". In fitting a neural network, backpropagation computes the gradient of the loss function with respect to the weights of the network for a single input–output example, and does so efficiently, unlike a naive direct computation of the gradient with respect to each weight individually. This efficiency makes it feasible to use gradient methods for training multilayer networks, updating weights to minimize loss; gradient descent, or variants such as stochastic gradient descent, are commonly used. The backpropagation algorithm works by computing the gradient of the loss function with respect to each weight by the chain rule, computing the gradient one layer at a time, iterating backward from the last layer to avoid redundant calculations of intermediate terms in the chain rule; this is an example of dynamic programming.
context_section_description: Calculating the partial derivative of the error with respect to a weight  is done using the chain rule twice:
  In the last factor of the right-hand side of the above, only one term in the sum, , depends on , so that
  If the neuron is in the first layer after the input layer,  is just .
  The derivative of the output of neuron  with respect to its input is simply the partial derivative of the activation function:
  which for the logistic activation function case is:
  This is the reason why backpropagation requires the activation function to be differentiable. (Nevertheless, the ReLU activation function, which is non-differentiable at 0, has become quite popular, e.g. in AlexNet)
  The first factor is straightforward to evaluate if the neuron is in the output layer, because then  and
  If the logistic function is used as activation and square error as loss function we can rewrite it as
  However, if  is in an arbitrary inner layer of the network, finding the derivative  with respect to  is less obvious.
  Considering  as a function with the inputs being all neurons  receiving input from neuron ,
  and taking the total derivative with respect to , a recursive expression for the derivative is obtained:
  Therefore, the derivative with respect to  can be calculated if all the derivatives with respect to the outputs  of the next layer – the ones closer to the output neuron – are known. [Note, if any of the neurons in set  were not connected to neuron , they would be independent of  and the corresponding partial derivative under the summation would vanish to 0.]
  Substituting Eq. 2, Eq. 3 Eq.4 and Eq. 5 in Eq. 1  we obtain:
  with
  if  is the logistic function, and the error is the square error:
  To update the weight  using gradient descent, one must choose a learning rate, . The change in weight needs to reflect the impact on  of an increase or decrease in . If , an increase in  increases ; conversely, if , an increase in  decreases . The new  is added to the old weight, and the product of the learning rate and the gradient, multiplied by  guarantees that  changes in a way that always decreases . In other words, in the equation immediately below,  always changes  in such a way that  is decreased:
caption: Diagram of an artificial neural network to illustrate the notation used here.
exif: {}