## 3 Flows of Signals and Gradients--Direct Derivation for a Small Network Example  
A good general understanding can be developed by studying small examples first. In this section, we provide a direct \"manual\" derivation of relevant formulas, working with an exemplative small network consisting of four layers, two of them convolutional. We believe this exercise helps abstract out the general formulas later on.  
The structure to consider is shown in Figure 3. Suppose it is meant to perform a classification task with 10 classes, based on \(64\times 64\) input images with 3 channels. The letters \(X,\alpha,\beta,\gamma,Y\) shall be used as names (to refer to layers) and also to represent the tensors storing actual values. In this section, we purposely use explicit numbers to define the sizes of tensors, filters, etc., rather than symbolic constants. Those explicit sizes shall thus be visible in the formulas, allowing the reader to clearly see their impact. Also purposely, we choose the sigmoid activation function for the output layer. We are aware that, e.g., the softmax activation would be a more natural choice for the classification task, but we prefer to postpone the discussion on softmax until Section 4.7.  
Below, we specify the names for collections of weights in tunable layers of the network. Convolutional weights are kept as 4D tensors, weights of the dense layer as a matrix, and weights related to free terms (also known as biases) as 1D vectors.  
\[W_{\alpha} =\left(w_{\alpha;a,b,p,q}\right)_{7\times 7\times 3\times 16}, W_{0,\alpha} =\left(w_{0,\alpha;q}\right)_{16}, \tag{5}\] \[W_{\beta} =\left(w_{\beta;c,d,p,q}\right)_{5\times 5\times 16\times 32}, W_{0,\beta} =\left(w_{0,\beta;r}\right)_{32},\] (6) \[W_{Y} =\left(w_{Y;e,f}\right)_{10\times 131\,027}, W_{0,Y} =\left(w_{0,Y;e}\right)_{10}. \tag{7}\]  
For clarity, we picked distinct letters for indexes pertaining to different pieces of information. For example, \(q\) is the index of an output channel (kernel) produced via layer \(\alpha\) and simultaneously the index of an input channel from the perspective of layer \(\beta\). Those letters are written in subscripts after a layer name and a semicolon. Additionally, we use the prefix 0 in front of a layer name to denote the collection of its biases.  
As regards activation functions in our example, we have \(\phi(\beta_{j,k,r})=[\beta_{j,k,r}>0]\cdot\beta_{j,k,r}\) since, for layer \(\beta\), the ReLU activation is chosen; then, \(\phi(\alpha_{j,k,q})=\alpha_{j,k,q}\), and \(\phi(\alpha_{j,k,q})/\alpha_{j,k,q}=1\), since no activation is declared for layer \(\alpha\), and finally, \(\phi(y_{e})=1/(1+\exp(-y_{e}))\)--the sigmoid function for layer \(Y\).  
In the following subsections, we discuss forward and backward computations for the described network. We do so layer by layer, writing first the more natural scalar-form formulas and then providing their tensor-form counterparts. Please note that at the implementation level, tensor formulas are typically well suited for CPU computations based on efficient libraries for algebra (e.g., BLAS and LAPACK underlying Python's numpy), whereas scalar formulas are typically better suited to (or more easily translatable to GPU computations.  
Figure 3: Example of a simple neural network for 10-class classification.