## 4 General Formulas For Common Layer Types
### Softmax Activation  
Softmax activations play an important role since their outcomes can be interpreted as probability distributions. They are commonly imposed on final output layers in classification tasks. The crucial property of the softmax function is that it depends not only on one weighted sum incoming to the neuron it activates but on all weighted sums computed via the layer. In consequence, a certain adjustment has to be made in the scheme of backward computations--element-wise multiplication by a vector of activation derivatives is no longer possible. But, as we show below, this adjustment can also be seen through the prism of main Rule 1 if softmax computations are treated as a two-staged process.  
Let us continue the notation from the preceding subsection, suppose that layer \(X\) is now equipped with the softmax activation with individual outputs computed as  
\[\phi(x_{k}):=\exp(x_{k})\Big{/}\sum_{0\leqslant z<N}\exp(x_{z}), 0\leqslant k<N. \tag{55}\]  
One should realize that a change in a single \(x_{z}\) signal causes changes in all \(\phi(x_{k})\) values, \(0\leqslant k<N\), because of the denominator, and thereby further changes in all neurons of layer \(\gamma\). Therefore, to suitably propagate back the errors \(\delta_{\gamma}\) onto \(\delta_{X}\), it is convenient to think of an intermediate stage of errors that we denote as \(\delta_{\phi}(X)\); see Figure 7.  
Partial derivatives \(\partial\phi(x_{z})/\partial x_{k}\) express quantitatively how strongly a \(\phi(x_{z})\) is affected by a change in \(x_{k}\). Hence, one can write the following formulas for backward propagation by applying Rule 1 twice. In the second case, \(\partial\phi(x_{z})/\partial x_{k}\) are treated as artificial weights of outgoing connections.  
\[\delta_{\phi(X);k} :=\sum_{0\leqslant j<M}w_{Y;j,k}\cdot\delta_{Y;j}, 0\leqslant k<N, \tag{56}\] \[\delta_{X;k} :=\sum_{0\leqslant z<N}\frac{\partial\phi(x_{z})}{\partial x_{k}} \cdot\delta_{\phi(X);z}, 0\leqslant k<N. \tag{57}\]  
Figure 7: Illustration of two stages of backward computations for a dense layer using a softmax activation function.  
All partial derivatives of the softmax function (or, in fact, any other) can be gathered into a matrix of the form  
\[D\phi(X):=\begin{pmatrix}\frac{\frac{\partial\phi(x_{0})}{\partial x_{0}}}{ \partial x_{0}}&\frac{\partial\phi(x_{1})}{\partial x_{0}}&\dots&\frac{\partial \phi(x_{M-1})}{\partial x_{0}}\\ \frac{\partial\phi(x_{0})}{\partial x_{1}}&\frac{\partial\phi(x_{1})}{ \partial x_{1}}&\dots&\frac{\partial\phi(x_{M-1})}{\partial x_{1}}\\ \dots&\ddots&\dots\\ \frac{\partial\phi(x_{0})}{\partial x_{M}}&\frac{\partial\phi(x_{1})}{ \partial x_{M}}&\dots&\frac{\partial\phi(x_{M-1})}{\partial x_{M-1}}\end{pmatrix} \tag{58}\]  
and applied in the following general formula for backward propagation through a dense layer:  
\[\delta_{X}:=D\phi(X)\cdot W_{Y}{}^{T}\cdot\delta_{Y}. \tag{59}\]  
Please note that Formula (54) is a special case of (59) for activation functions depending on a single argument.  
As regards actual derivatives for the softmax function, it is easy to check that, after some algebra, one obtains the following:  
\[\frac{\partial\phi(x_{z})}{\partial x_{k}}=\begin{cases}\phi(x_{z})(1-\phi(x _{z})),&\text{for}\,z=k;\\ -\phi(x_{z})\phi(x_{k}),&\text{for}\,z\
eq k.\end{cases} \tag{60}\]  
This means that explicit analytic expressions are not needed at the implementation level, and one can reuse neurons' outcomes from the forward pass to compute the derivatives fast during the backward pass. A similar property is true for sigmoid activation functions, where \(\phi^{\prime}(x)=\phi(x)(1-\phi(x))\), which agrees with softmax partial derivatives for \(z=k\).  
In short, (60) is equal to \(\partial\phi(x_{z})/\partial x_{k}=\phi(x_{z})([z=k]-\phi(x_{k}))\), and in the matrix form, it becomes:  
\[D\phi(X)=I_{N}\cdot\phi(X)-\phi(X)\cdot\phi^{T}(X). \tag{61}\]