## 3 Flows of Signals and Gradients--Direct Derivation for a Small Network Example
### Backward Pass  
Suppose that categorical cross-entropy is selected as the loss function to fit our network and that \(Y^{*}\) stands for a one-hot encoded vector representing the output wanted for the given training example. The loss between \(Y\) and \(Y^{*}\) is  
\[l(Y,Y^{*})=-\sum_{0\leqslant e<10}y_{e}^{*}\log\phi(y_{e}). \tag{16}\]  
In accordance with the chain rule, loss derivatives with respect to \(w_{Y;e,f}\) weights can be calculated as the following product:  
\[\frac{\partial l}{\partial w_{Y;e,f}}=\frac{\partial l}{\partial\phi(y_{e})} \cdot\frac{\partial\phi(y_{e})}{\partial y_{e}}\cdot\frac{\partial y_{e}}{ \partial w_{Y;e,f}}=-\frac{y_{e}^{*}}{\phi(y_{e})}\cdot\phi(y_{e})(1-\phi(y_{ e}))\cdot\phi(\gamma_{f}), \tag{17}\]  
where the second line applies the actual derivatives. Now, we start deriving the backpropagation algorithm for our network by writing down partial derivatives of the loss as the following column vector:  
\[\delta_{l}:=\bigg{(}\frac{\partial l}{\partial\phi(y_{0})}\quad\frac{ \partial l}{\partial\phi(y_{1})}\quad\dots\quad\frac{\partial l}{\partial \phi(y_{9})}\bigg{)}^{T}. \tag{18}\]  
Even though the loss function is not an actual layer in the network, we purposely denote the above vector as \(\delta_{l}\) and treat it as our initial error quantity from which the induction ought to be started. Having \(\delta_{l}\), one can define errors \(\delta_{Y}\) for the proper last layer using Rule 1 (see page 8):  
\[\delta_{Y}:=\phi^{\prime}(Y)\circ(I_{10}\cdot\delta_{l})\] \[\qquad=\bigg{(}\frac{\partial\phi(y_{0})}{\partial y_{0}}\cdot \frac{\partial l}{\partial\phi(y_{0})}\ \frac{\partial\phi(y_{1})}{\partial y_{1}}\cdot\frac{ \partial l}{\partial\phi(y_{1})}\ \dots\ \frac{\partial\phi(y_{9})}{\partial y_{9}}\cdot\frac{ \partial l}{\partial\phi(y_{9})}\bigg{)}^{T}, \tag{19}\]  
where each entry, for our example, is \(\delta_{Y;e}=\phi(y_{e})(1-\phi(y_{e}))\cdot(-y_{e}^{*}/\phi(y_{e})),\ 0 \leqslant e<10\). We explain that the identity matrix \(I_{10}\), written purposely in the first line of (19), can be seen as a matrix of weights for outgoing connections. Every neuron in the output layer, \(Y\), can be thought of as having exactly one outgoing connection, with weight \(1\), to a corresponding virtual loss neuron. Obviously, the presence of an identity matrix does not change anything mathematically, but we want to keep the writing comprehensive and consistent with the inductive Rule 1. Having computed \(\delta_{Y}\), we can compute the gradient for weights in the last layer with the use of Rule 2. It yields  
\[\
abla_{Y}:=\delta_{Y}\cdot\phi(\gamma)^{T}, \tag{20}\] \[\
abla_{0,Y}:=\delta_{Y}\cdot\mathbf{1}_{10\times 1}, \tag{21}\]  
respectively for proper connections and for biases. We remark that, with (20) and (21) computed, corrections of weights could be potentially performed. For example, using the simple SGD, one would have \(W_{Y}:=W_{Y}-\eta\ \
abla_{Y}\), and \(W_{0,Y}:=W_{0,Y}-\eta\ \
abla_{0,Y}\), where \(\eta>0\) stands for the learning rate. However, correction steps are typically withheld until the complete backward pass through all the layers is finished. We now move to the flattening layer, \(\gamma\). Again, in accordance with Rule 1, a particular \(\delta_{\gamma_{7};f}\) must take into account its own activation derivative, and subsequent \(\delta\)s reached via outgoing connections. In the scalar form, this yields  
\[\delta_{\gamma;f}:=\underbrace{\frac{\partial\phi(\gamma_{f})}{\partial\gamma_{ f}}}_{1}\cdot\sum_{0<\varepsilon<10}w_{\gamma;e,f}\cdot\delta_{\gamma_{7};e}, \tag{22}\]  
for \(0\leqslant f<131\,072\), which, in the tensor form, is equivalent to  
\[\delta_{\gamma}:=\phi^{\prime}(\gamma)\circ(W_{\gamma}{}^{T}\cdot\delta_{ \gamma})=\frac{1}{{}_{131\,072}}\circ(W_{\gamma}{}^{T}\cdot\delta_{\gamma}). \tag{23}\]  
Again, the initial multiplication by a column vector of ones is presented for the sake of consistency with Rule 1. This vector represents derivatives of the activation function (none declared).