## 4 Core Algorithm and Implementation
### Structure Design of the Fully Connected Layer
#### 4.2.1 Fully Connected Layer Multiplication Structure  
Fully connected forward propagation is expressed as:  
\[Z^{l}=W^{l}*a^{l-1} \tag{13}\]  
where \(W^{l}\) is the fully connected weight matrix of the \(l\)-th layer, and \(a^{l-1}\) is the column vector output after the fully connected activation of the \(l-1\)-th layer; and \(Z^{l}\) is the column vector output before the full connection activation of the \(l\)-th layer. The realization of fully connected matrix multiplication is as shown in Algorithm 4.  
```
h: Fully connected input vector dimension h_out: Fully connected output vector dimension input_matrix: Fully connected input vectors para_layer: Fully connected weight matrix out_matrix: Fully connected output vectors
1. for(int i = 0; j < h_out; j++){
2. output_matrix[j] = 0;
3. for(int i = 0; i < h; i++)
4. output_matrix[j] += input_matrix[i] \(\times\) para_layer[i \(\times\) h_out + i]; ]
```  
**Algorithm 4** MatrixMultiPLy.  
Given the output loss \(\frac{\partial C}{\partial Z^{l+1}}\) of the fully connected \(l+1\)-th layer, calculate the output loss \(\frac{\partial C}{\partial Z^{l}}\) after activation of the \(l\)-th layer:  
\[\frac{\partial C}{\partial d^{l}}=\frac{\partial C}{\partial Z^{l+1}}*\frac{ \partial Z^{l+1}}{\partial d^{l}} \tag{14}\]  
By substituting the fully connected forward propagation (Equation (12)), we can obtain:  
\[\frac{\partial C}{\partial d^{l}}=\frac{\partial C}{\partial Z^{l+1}}*\frac{ \partial W^{l+1}*a^{l}}{\partial d^{l}} \tag{15}\]  
Figure 8: Data propagation at the full connection layer.  
According to the matrix derivative rule, we can obtain:  
\[\frac{\partial\mathcal{C}}{\partial a^{l}}=\left(W^{l+1}\right)^{T}*\frac{ \partial\mathcal{C}}{\partial Z^{l+1}} \tag{16}\]  
The algorithm is implemented as shown in Algorithm 5.  
```
w: Fully connected weight matrix width h: Fully connected weight matrix height input_matrix: Fully connected weight matrix grad: Fully connected output gradient out_matrix: Fully connected input gradient
1. for(int i = 0; i < w; i++){  output_matrix[i] = 0;// Gradient clear, easy to add
3. for(int j = 0; j < h; j++)
4. output_matrix[i] += input_matrix[i \(\times\) h + j] \(\times\) grad[j]; }
```  
**Algorithm 5** CalculateMatrixGrad.  
Calculate the output loss \(\frac{\partial\mathcal{C}}{\partial Z^{l}}\) of the l-layer fully connected weight matrix given the output loss \(\frac{\partial\mathcal{C}}{\partial W^{l}}\) of the l-layer fully connected:  
\[\frac{\partial\mathcal{C}}{\partial W^{l}}=\frac{\partial\mathcal{C}}{ \partial Z^{l}}*\frac{\partial Z^{l}}{\partial w^{l}} \tag{17}\]  
By substituting the fully connected forward propagation (Equation (13)), we can obtain:  
\[\frac{\partial\mathcal{C}}{\partial W^{l}}=\frac{\partial\mathcal{C}}{ \partial Z^{l}}*\frac{\partial W^{l}*a^{l-1}}{\partial w^{l}} \tag{18}\]  
According to the matrix derivative rule, we can obtain:  
\[\frac{\partial\mathcal{C}}{\partial W^{l}}=\frac{\partial\mathcal{C}}{ \partial Z^{l}}*\left(d^{l-1}\right)^{T} \tag{19}\]  
The algorithm is implemented as shown in Algorithm 6.  
```
w: Fully connected weight matrix width h: Fully connected weight matrix height input_matrix: Fully connected input gradient grad: Fully connected output gradient grad: Fully connected weight matrix gradient
1. for(int i = 0; i < w; i++)
2. for(int j = 0; j < h; j++)
3. \(\text{rgrad[i }\times\) h + j] = input_matrix[i] \(\times\) grad[j];
```  
**Algorithm 6** MatrixBackPropagationMultiPLy.