# Hands-On Fundamentals of 1D Convolutional Neural Networks--A Tutorial for Beginner Users
## 5 Backpropagation
Defining \(\Delta y_{out}\) as a vector \(2\times 1\) vector  
\[\Delta y_{out}(i)=(y_{out}(i)-y_{t}(i))\cdot y_{out}(i)\cdot(1-y_{out}(i)) \tag{15}\]  
the gradient \(\Delta W\) can be rewritten as  
\[\Delta W=\Delta y_{out}\times f^{T}. \tag{16}\]  
where \" \(\times\) \" refers to the matrix product, and \(f^{T}\) represents the transpose of the vector \(f\), which is a \(1\times 18\) vector. The same calculation can be applied for the bias gradient:  
\[ \begin{split}\Delta b(i)&=\frac{\partial L}{ \partial b(i)}\\ &=\frac{\partial L}{\partial y_{out}(i)}\cdot\frac{\partial y_{ out}(i)}{\partial b(i)}\\ &=(y_{out}(i)-y_{t}(i))\cdot\frac{\partial}{\partial b(i)} \sigma\biggl{(}\sum_{j=1}^{18}W(i,j)f(j)+b(i)\biggr{)}\\ &=(y_{out}(i)-y_{t}(i))\cdot y_{out}(i)\cdot(1-y_{out}(i))\\ &=\Delta y_{out}(i).\end{split}  \tag{17}\]  
Before updating the parameters in \(CL_{2}\), it is necessary to calculate the gradient of the vector \(f\). Each entry of \(\Delta f\) is then represented by  
\[ \begin{split}\Delta f(j)&=\frac{\partial L}{ \partial f(j)}\\ &=\sum_{i=1}^{2}\frac{\partial L}{\partial y_{out}(i)}\cdot\frac{ \partial y_{out}(i)}{\partial f(j)}\\ &=\sum_{i=1}^{2}(y_{out}(i)-y_{t}(i))\cdot y_{out}(i)\cdot(1-y_{ out}(i))\cdot W(i,j)\\ &=\sum_{i=1}^{2}\Delta y_{out}(i)W(i,j)\end{split}  \tag{18}\]  
and \(\Delta f\) can now be written as  
\[\Delta f=W^{T}\times\Delta y_{out}. \tag{19}\]  
During the feedforward step, the vector \(f\) results from \(S^{(2)}\) reshaping. Here, in the backpropagation step, the gradient \(\Delta f\) is reshaped back to obtain \(\Delta S^{(2)}\), as schematically depicted in Figure 6.  
In the \(CL_{2}\), an un-pooling operation should be performed to up-sample the compressed data \(\Delta S^{(2)}\) and obtain \(\Delta C_{\sigma}^{(2)}\). To avoid confusion, great attention is now needed. If average up-sampling was used in feedforward, the backpropagation error is multiplied by \(1/SS^{(2)}\) and uniformly assigned to all the elements in the pooling block (Figure 7). In this case, each column of \(\Delta C^{(2)}\) can be written as  
\[\Delta C^{(2)}_{j\sigma}(i_{u})=\frac{1}{SS^{(2)}}\Delta S^{(2)}_{j}\Big{(}\Big{[} i_{u}/SS^{(2)}\Big{]}\Big{)} \tag{20}\]  
where the index \(i\) of \(\Delta S^{(2)}_{j}\) runs from \(1\) to \(n=3\), \(i_{u}\) runs from \(1\) to \(6=SS^{(2)}n\), and finally \(\lceil x\rceil\) represents the ceiling function. As in previous instances, in the following, \(i\) will denote the index \(i_{u}\). On the other hand, if max-pooling was performed during the feedforward step, the error is assigned to the location it originated from, and zero is assigned to all the other elements. This implies that during the feedforward step, the index of the \"winning unit\" should be stored and reused in the backpropagation step. This aspect may not be evident when using pre-packaged solutions but should be taken into account when developing new CNN libraries. Once \(\Delta C^{(2)}_{\sigma}\) has been computed, it is possible to update the filter value in \(CL_{2}\). This means calculating each element of the third-order tensor \(\Delta K^{(2)}\), with \(j=1,\ldots p=4\), \(l=1,\ldots q=6\), and \(u=1,\ldots 3\), we obtain  
Figure 6: The vector \(f\) (\(18\times 1\)) is reshaped back in order to obtain \(\Delta S^{(2)}\) (\(3\times 6\)). Figure 7: Example of an up-sampling operation. The input is the \(2\times 1\) array on the left. Down-sampling produces a 1D array based on the average or maximum of the input. During up-sampling, the 1D array is split into a new \(2\times 1\) array. If averaging is used, the output is filled by evenly distributing the output value from down-sampling, in this case, dividing by \(2\). If the max operation is used, the output value is placed where the original maximum value occurred.