## 2 Background
### Backpropagation  
In this section, we turn our attention to automatic differentiation and the backpropagation algorithm. Obviously, various structural novelties that appeared throughout the history--weight sharing, convolutions, pooling, ReLUs, softmax activation, dropout, etc.--and possibly others still to come in the future, need to be handled properly during the computations of derivatives, which allow for tuning network parameters. Therefore, apart from being correct and efficient, the algorithm for that purpose needs to be (and is) universal. With that universality in mind, here, we discuss the general principles that underlie backpropagation. The usual goal is to train (using data examples) a neural network that can be seen as a large _graph of connections_ with tunable parameters called _weights_. Commonly, the connections are organized as a layered structure; see, e.g., Figure 2. The training is carried out through some variant of the gradient descent in order to minimize a certain error function for the whole training data set with respect to weights. That error is, in turn, defined by the so-called _loss_ function at the level of a single training pair, \((\mathbf{x},y^{*})\)--categorical cross-entropy and squared loss being the common choices for loss functions in classification and regression tasks, respectively. For the large majority of networks (graphs of connections), automatic differentiation can be constructed from two rules well known from calculus: (i) the chain rule--stating that, for a composition of two differentiable functions, the derivative of the composition is equal to the derivative of the outer function (calculated with respect to the inner one) multiplied by the derivative of the inner function; (ii) the sum rule--stating that a derivative of a sum of functions is the sum of their derivatives. In practice, the rules imply that intermediate derivatives leading towards a particular weight along a _single_ path of connections must be _multiplied_, whereas the ones along _alternate_ paths must be _summed_. Although a derivative for any individual weight could be computed separately via the above approach, doing so would be inefficient because of unnecessary recomputations of many intermediate values. Therefore, one can do better by applying an _induction_. For a fixed training example (or a batch of examples), the induction proceeds from the back of the network towards the front, layer by layer, and computes derivatives specific to the given layer--components of the final gradient--based on values already computed. To achieve this, one introduces an auxiliary quantity, traditionally named \(\delta\) (we skip its subscripts for a moment), associated with every neuron and computed inductively. That quantity is interpreted as an \"error\", but more accurately, it should be seen as a partial contribution to the final loss derivative that takes place from the point of the given neuron onwards in the network. We must clarify that \(\delta\) values themselves are _not_ equal to loss derivatives. To turn them into derivatives, an additional multiplication by inputs of the given layer is necessary. Let \(l\) denote the loss function, and suppose that we are about to perform backward computations for the \(i\)-th layer, having already performed the computations for layers \(i+1,i+2,\ldots\). Suppose also that the layer is equipped with a collection (a tensor) of weights, denoted here as \(W_{i}\). The algorithmic goal is to be able to compute \(\partial l/\partial w\) for any \(w\in W_{i}\), based on error quantities \(\delta_{i}\). The key argument to realize is that \(w\) can affect the loss of the whole network only through its direct effect on the very next layer, \(i+1\). Therefore, to compute \(\delta_{i}\)--the collection of error quantities for the current layer--one needs to know only \(\delta_{i+1}\) and \(W_{i+1}\), i.e., direct subsequent errors and weights of outgoing connections through which these errors are reached, respectively. Error quantities for \(i+2,i+3,\ldots\) are not needed.