# Efficient FPGA Implementation of an ANN-Based Demapper Using Cross-Layer Analysis
## 4 Hardware Implementation
### Hardware Architecture
#### 4.2.2 Training Module  
The hardware design of the training module is illustrated in Figure 6. As it is shown, it is divided into the forward-pass to calculate the output and the backward-pass to compute the gradient and update the weights. Analogically to the inference module, it operates in a pipelined fashion, starting from the first layer in the forward-pass and from the last layer in the backward-pass. The forward-pass is similar to the inference module with the sole difference that the feature maps between the layers are stored to calculate the gradients during backpropagation. The architecture includes an adjustable batch size, corresponding to the number of samples that are forward-propagated before the update of weights is applied, which is kept very small for the low-latency AE implementation. The backpropagation itself starts by calculating the error term \(\delta_{j}\) of the loss function depending on the current output \(a_{j}\) and the corresponding label \(y_{j}\), as:  
\[\delta_{j}=a_{j}-y_{j} \tag{3}\]  
Based on this error, the error for each fully connected layer followed by a ReLU layer, up until the first one is calculated, determined by current weight and the error of the following layer, as:  
\[\delta_{j}=\underset{k}{\sum}\sigma_{j,k}w_{i,j}\ \ \ \text{with}\ \sigma_{j,k}= \begin{cases} \delta_{j,k}&\text{if}\ \text{fm}_{j,k}\geq 0\\ 0&\text{otherwise}\end{cases}  \tag{4}\]  
where \(k\) corresponds to the following layer's neuron, \(j\) corresponds to the current layer's neuron and \(i\) corresponds to the previous layer's neuron. This equation is implemented as multiplexer, followed by an adder tree, where parallelization can be adjusted by PE. Based on the error \(\delta_{j}\), the weights can be updated as:  
\[\Delta w_{i,j}=\eta\underset{k}{\sum}\sigma_{j,k}\text{fm}_{i,j}\ \ \ \text{with}\ \sigma_{j,k}= \begin{cases}\delta_{j,k}&\text{if}\ \text{fm}_{j,k}\geq 0\\ 0&\text{otherwise}\end{cases}  \tag{5}\]  
where \(\eta\) refers to the learning rate, and \(fm\) to the feature map saved by the forward-pass. This is again implemented in hardware using an adder tree. After one batch is processed, all trainable parameters are individually updated by subtracting \(\Delta w_{i,j}\) from the corresponding weight \(w_{i,j}\). The calculation of the error term \(\delta_{j}\) and the update of the weights for a fully connected layer followed by a ReLU activation function is depicted on the right side of Figure 6. As shown, the error term is propagated between different layers using FIFOs while the weights are stored in block RAM (BRAM). The weights are partitioned in a way to allow accessing \(sind*pe\) weights in one clock cycle to not slow down the computation. To allow for different DOP of inference and training module, the weights are transformed corresponding to the partition scheme when exchanged between the modules. In general, the DOP can be calculated as the product of SIMD and PE, referring to input and output parallelism, as shown in Figure 6. In contrast to the inference module, the weights of the  
Figure 6: FPGA design of the training module. training module need to be quantized with up to 14 bits. The main reason for the higher bit-width is that for each training iteration, only a small adaption of the weights is performed; so, higher precision is needed to track this update of the weights. After backpropagation is applied for multiple iterations, the difference becomes significant enough to be also representable by the lower bit-width used during inference.