## 4 Core Algorithm and Implementation
### Structure Design of the Fully Connected Layer
#### 4.2.2 Activation Function Structure at the Full Connection Layer  
Forward propagation of the activation function is expressed as:  
\[a^{l}=\sigma\left(Z^{l}\right) \tag{20}\]  
where \(\sigma\left(\right)\) represents the activation function, \(Z^{l}\) is the input of the activation function at the \(l\)-th layer, and \(a^{l}\) is the output of the activation function at the \(l\)-th layer. LeakyRelu replaces Relu in the activation function to avoid gradient dispersion. The algorithm is implemented as shown in Algorithm 7.  
Given the output loss \(\frac{\partial\mathcal{C}}{\partial x^{I}}\) of \(l\)-th layer after activation, calculate the output loss \(\frac{\partial\mathcal{C}}{\partial Z^{I}}\) of fully connected \(l\)-th layer:  
\[\frac{\partial\mathcal{C}}{\partial Z^{I}}=\frac{\partial\mathcal{C}}{\partial x ^{I}}*\frac{\partial x^{I}}{\partial Z^{I}}=\frac{\partial\mathcal{C}}{ \partial x^{I}}*\sigma^{\prime}\Big{(}Z^{l}\Big{)} \tag{21}\]  
The algorithm is implemented as shown in Algorithm 8.  
```
h: The height of the column vector to be activated input_matrix: Activate the rear column vector grid: Gradient after activation output_matrix: The gradient before activation
1. for(int i = 0; i < w; i++)
2. if(input_matrix[i] > 0)
3. output_matrix[i] = 1 \(\times\) grad[i];
4. else
5. Output_matrix[I] = 0.05 \(\times\) grad [I];
```  
**Algorithm 7** Relu.  
For the fully connected layer, the input and output feature graphs and gradients are one-dimensional vectors, and the weights and gradients are two-dimensional vectors. According to Equations (13) and (20), when the forward propagation of the fully connected module is realized, the output result of the fully connected module can be obtained by activating the function after the weighted operation of the feature graph and the weight. According to Equations (16) and (21), when the fully connected model is backpropagated, the output gradient of the backpropagation can be obtained after the input gradient is weighted by the derivative of the activation function and the transpose of the weight matrix. According to Equation (19), when the fully connected module is implemented to update the weight matrix, the gradient of the weight matrix can be obtained after the weighting operation of the transpose of the input gradient and the input feature graph, which can update the weight matrix combined with the learning rate. By combining Equations (13), (16) and (19)-(21), the forward propagation and backpropagation operation models of the fully connected layer are obtained. The most basic hardware accelerator structure of the fully connected module is shown in Figure 9.  
Using the Unroll command, you can transform the fully connected layer software model represented by loops into an efficient hardware accelerator architecture. Table 1 shows the hardware IP resource usage corresponding to the LeNet-3.3 network.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **BRAM\_18K** & **DSP48E** & **FF** & **LUT** \\ \hline Total & 419 & 134 & 20,962 & 36,190 \\ Utilization (\%) & 96 & 37 & 14 & 51 \\ \hline \hline \end{tabular}
\end{table}
Table 1: LeNet-3.3 IP core resource usage details.