## 8 Implementation and Experiments
### Automatic Differentiation (Backward Computations) for Common Layer Types  
_Flattening layers_ (class Flatten): We purposely start this subsection with the simplest layer type as far as differentiation is concerned--namely the flattening layer. We remind the reader that the chain rule (Rule 1) that computes the \(\delta_{X}\) tensor within the backpropagation induction says to apply the derivative of the activation function multiplied by the weighted sum of subsequent \(\delta_{Y}\) values taken over outgoing connections. In the case of flattening layers, each neuron--a signal in the input tensor \(X\) (of shape \(H\times V\times C\))--has exactly one outgoing connection to its counterpart in the flattened vector \(Y\). Therefore, the appropriate inductive formula (we repeat Formula (47) from Section 4) is as follows:  
\[\delta_{X}:=\phi^{\prime}(X)\circ\underset{H\times V\times C}{\text{R}}(I_{H \cdot V\cdot C}\cdot\delta_{Y})=\phi^{\prime}(X)\circ\underset{H\times V\times C}{ \text{R}}\delta_{Y}.\]  
Backward computations associated with this formula are reduced to a suitable call of the reshape function, preceded by a call of self.activation_d_() to compute the derivative of the activation function (if any activation was imposed by the user). The source code in Figure 16 presents those simple computations. _Dropout layers_ (class Dropout): In the case of dropout layers, automatic differentiation is also quite simple. Below, we repeat Formulas (48) and (49) for both the forward and backward computations of those layers. We remind the reader that \(r^{*}\) represents the imposed rate of signals to be dropped (zeroed) in the current pass of computations, whereas  
Figure 16: Automatic differentiation for class Flatten. Figure 15: Forward and backward passes for SequentialClassifier. the tensor \(R=(r_{j,k,q})\), of shape \(H\times V\times C\), has entries drawn randomly from a standard uniform distribution, \(r_{j,k,q}\sim U(0,1)\). \[Y :=\frac{1}{1-r^{*}}\cdot[R>r^{*}]\circ\phi(X).\] \[\delta_{X} :=\phi^{\prime}(X)\circ\frac{1}{1-r^{*}}\cdot[R>r^{*}]\circ\delta_ {Y}.\]  
At the implementation level, it is useful to memorize, during the forward pass, the tensor of random entries, \([R>r^{*}]\) (the ones to be preserved), together with their multipliers, \(1/(1-r^{*})\), and then to make use of it during the backward pass. In the code from Figure 17, that combined information is memorized as the attribute self.coeffs_. As was the case for the flattening layers, also here, in the dropout layers, each \"neuron\" can be regarded as having exactly one outgoing connection. Dense layers (class Dense): In contrast to flattening and dropout layers, dense layers do include tunable parameters--weights. Therefore, the backward pass for this layer type involves both the inductive computations of \(\delta\) values (Rule 1) and the computations of the gradient (Rule 2). We remind the reader of the necessary Formulas (52)-(54) from Section 4 below. \[\
abla_{Y} :=\delta_{Y}\cdot\phi(X)^{T}.\] \[\
abla_{0,Y} :=\delta_{Y}\cdot\underbrace{1}_{N\times 1}.\] \[\delta_{X} :=\phi^{\prime}(X)\circ W_{Y}^{T}\cdot\delta_{Y}.\]  
The code excerpts shown in Figure 18 are responsible for the above computations. It is also worth recalling (as noted in Section 8.1) that, inside hmdl tensors, the first dimension is always related to the data example index within the current _batch_. Therefore, in the code below, the attributes delta_backward_input_, delta_, and delta_backward_output_ are two-dimensional arrays, where the first dimension pertains to data examples and the second to neurons within the given layer. Owing to that fact, suitable summations over the current batch needed to compute the self.gradient_ are carried out by means of the.dot product, i.e., the standard algebraic matrix multiplication (see the function do_backward_numpy). Figure 17: Automatic differentiation for class Dropout.