## 8 Implementation and Experiments
### Automatic Differentiation (Backward Computations) for Common Layer Types
_Max pooling layers_ (class MaxPool2D): In the hmdl project, there are three implementation variants for computations carried out within the max pooling layers. The first one is a \"pure\" numpy variant. The second is an accelerated version of the first, obtained by means of numba module and the ejit(nopython=True,...) decorator, allowing for low-level just-in-time compilation. The third variant also applies numba, but it performs GPU computations--it requires the ecuda.jit decorator over a suitably programmed kernel function in compliance with the CUDA computational model. For clarity of presentation, below, we show the source codes only for the simplest numpy-based variants of computations. Source codes for the two other variants are moved to Appendix A. Recall the notation and suitable formulas from Section 4; the pools are assumed to be of sizes \(S\times S\), and the single outputs, for all \(0\leqslant j<\lfloor H/S\rfloor\), \(0\leqslant k<\lfloor V/S\rfloor\), and \(0\leqslant q<C\), are computed as follows:  
\[y_{j,k,q}:=\max_{ \begin{subarray}{c}j\leqslant n<(j+1)S\\ \leqslant c<(k+1)S\end{subarray} }\phi(x_{u,p,q}).\]  
Again, it is useful to memorize, during the forward pass, the coordinates for which the maxima are attained, denoted by  
\[a^{*}_{j,k,q}:=\operatorname*{arg\,max}_{ \begin{subarray}{c}j\leqslant n<(j+ 1)S\\ \leqslant c<(k+1)S\end{subarray} }\phi(x_{u,p,q}), \tag{112}\]  
as they are needed at the stage of backward computations. Figure 19 presents the source codes responsible for forward computations in the max pooling layers in the simplest numpy-based variant. Figure 18: Automatic differentiation for class Dense. As regards the backward pass and automatic differentiation for this layer type, Formula (43), required for that purpose, is as follows:  
\[\delta_{Xu,v,q}:=\phi(x_{u,v,q})\cdot \begin{cases}\delta_{Y_{c}^{*}\lfloor u/S \rfloor,\lfloor v/S\rfloor},&\text{if}a^{*}_{\lfloor u/S\rfloor,\lfloor v/S \rfloor}=(u,v);\\ 0,&\text{otherwise.}\end{cases} \]  
Backward computations compliant with this formula are presented in Figure 20 (numpy-based variant). Please note how the attribute self.argmax_, memorized during the forward pass, is reused at this stage. We remind the reader that other computational variants are available in Appendix A.  
Convolutional layers (class Conv2D)In the hmdl project, there are seven implementation variants of forward computations for convolutional layers and four variants of backward computations. Since this subsection is on automatic differentiation, we discuss here only the backward variants. Forward variants and their efficiency are discussed in a separate section, Section 8.4. As regards backward computations in convolutional layers, the variants available in the hmdl project are numpy_gemm, numba_jit_gemm, numba_cuda_direct, and numba_cuda_tiles. For clarity of presentation, here, we present source codes only for the numpy_gemm variant; the remaining variants are moved to Appendix B.  
Figure 19: Forward computations for class MaxPool2D: the simplest numpy-based variant. We start by reminding the reader of Formula (38) (from Section 4) pertaining to the computation of gradient for convolutions in compliance with the GEMM approach:  
\[\
abla_{Y}:=\underset{0\leq q<K}{\operatorname{\mathbb{R}}}\operatorname{ \mathbb{R}}_{\forall F\times F\times C}\left(\left(\underset{1\times k\cdot W}{ \operatorname{\mathbb{R}}_{Y}}\delta_{Y_{\cdot}\cdot\cdot\cdot\cdot\cdot}\right) \left(\underset{0\leq p<C}{\operatorname{\mathbb{S}}}\underset{1\cdot H\cdot W \times 1}{\operatorname{\mathbb{S}}}\phi\left(X_{\underset{p+i}{p+i}{p+i}} \right)\right)\right),\] \[\
abla_{0,Y}:=\underset{0\leq q<K}{\operatorname{\mathbb{S}}} \left(\left(\underset{1\times H\cdot W}{\operatorname{\mathbb{R}}_{Y_{\cdot} \cdot\cdot\cdot\cdot\cdot}}\delta_{Y_{\cdot}\cdot\cdot\cdot\cdot}\right)\cdot \underset{H\cdot W\times 1}{\operatorname{\mathbb{S}}}\right),\]  
Figure 21, shown below, presents the source code responsible for those computations.