# Implementation of the SoftMax Activation for Reconfigurable Neural Network Hardware Accelerators
## 5 Experimental Results and Discussion  
The characteristics of the proposed SoftMax implementation were evaluated through simulations on Field-Programmable Gate Arrays (FPGA). FPGAs are widely used in the research and development of state-of-the-art computing systems, and numerous useful software tools are available [6; 7; 23]. The models were implemented in the form of digital design modules using Verilog HDL and Quartus Prime 20.1.0 software. The modules operate on 16-bit fixed-point numbers (8 fractional bits). All the simulations were performed on a Cyclone V 5 CGXFC9E7F35C8 device with disabled Digital Signal Processing (DSP) blocks. To avoid unintentional optimizations, the \"synthesis keep\" directive was applied. We evaluated the resource utilization and performance of the proposed models. In this paper, resource utilization refers to the number of FPGA logic elements (LEs) required to implement the module. The Quartus software provides this value after a successful compilation. The model performance was measured as the largest processing delay. To evaluate this value, the Timing Analyzer tool (included in the Quartus software pack) was used. This tool analyzes all available paths between the inputs and the outputs, computes their signal propagation delay, and determines the longest one. Due to the large number of interconnections between processing elements, Quartus treats them as combinational loops. To tackle this problem, some unused connections were removed during the timing simulation. The simulation results are presented in Table 2. As can be seen, the proposed models have high performance. The total processing delay of the SoftMax activation for three inputs is 43 ns. Exponential blocks are processed in 14 ns. Since these blocks work in parallel, their total delay is independent of the number of inputs. The processing delay of the other stages (applying the shift and computing the sum) linearly depends on the number of inputs. At the same time, the models consume a large amount of FPGA resources. The entire implementation of the SoftMax activation for three inputs utilizes about 55,000 LEs, including 15,000 LEs per exponential block. An explanation of these results will be provided further. The achieved results were compared with other implementations lacking the reconfiguration capability (Table 3). Figure 10: SoftMax activation in the RCE. As is evident from Table 3, the proposed in this paper implementation provides the comparable throughput (about 1.12 Gbps) despite the lower clock frequency (about 23.3 MHz). In other words, it processes the commensurate amount of input data per unit of time, performing more calculations per clock cycle. However, the FPGA resource utilization is noticeably high (about 55,000 LEs compared to 18,000 of the counterpart). Thus, the proposed model consumes more space (in terms of a semiconductor area or the number of transistors) but does not provide a significant increase in performance. The results presented in Tables 2 and 3 can be explained by the following reasons:  
* Specialized computing devices are capable of achieving the best results for specific tasks due to particular optimization. Reconfigurable devices, by contrast, must sacrifice some properties to provide acceptable results for a wide variety of tasks. Our implementation of a reconfigurable accelerator sacrifices resource utilization (number of logic elements) to provide comparable performance. * The proposed algorithms are spatially distributed. Different parts of the computing structure calculate different branches or stages of these algorithms. Spatial distribution leads to better performance (for some algorithms) at the expense of increased resource consumption;
* To provide the reconfiguration ability, each processing element supports the entire set of operations, although it performs only one of them at any given moment;
* The PEs must support simple reusable operations to be small and versatile.