# Implementation of the SoftMax Activation for Reconfigurable Neural Network Hardware Accelerators
## 4 Implementation of the SoftMax Activation in RCE
### Current State of Hardware Implementation of the SoftMax Activation  
According to Equation (1), the SoftMax activation incorporates the exponential function. This is the non-linear fast-growing function (Figure 6). Since the hardware implementation of this function is resource-intensive, researchers have proposed various approaches to optimize it. In papers [29; 30; 31; 32], the exponential function is implemented by storing its points in an internal memory of FPGA (look-up table, LUT). This is a simple and effective solution. However, to achieve sufficient accuracy, the number of stored points must be large enough. As a result, this solution is convenient for sequential processing only when the device computes its inputs one by one. To perform the calculations in parallel, each branch needs its own table of points, leading to high memory consumption. To reduce the memory footprint, in [29], the power of \(e\) is decomposed into digits. The exponential function values for these points are stored in the device memory. These values are then multiplied to produce the final result. More promising solutions utilize bitwise shift operation [13; 33]. They propose replacing \(e^{x}\) with \(2^{y+z}\), where \(y\) and \(z\) are the integer and fractional parts, respectively. Since the bitwise shift has a well-known efficient hardware implementation, the solution provides good results. Nevertheless, performing the bitwise shift operation requires several clock cycles, leading to difficulties for asynchronous devices. The papers [22; 30] propose a similar solution based on the Taylor expansion and bitwise shift. Unfortunately, both presented solutions cannot be efficiently implemented in an RCE. The LUT-based implementations are not applicable due to the homogeneity of RCE--all its processing elements must be identical. Storing the entire look-up table of the exponential function in each PE results in low efficiency and high memory consumption. Furthermore, the stored points are applicable for the exponential function only and cannot be reused for other tasks. This fact violates the key design principle of RCE processing elements that imply high reusability of operations in an operation set. The implementations based on bitwise shift cannot be employed in an RCE for several reasons. As mentioned before, the bitwise shift operation implies synchronous data processing and takes several clock cycles. Nevertheless, the RCE processing elements work asynchronously and do not rely on the clock cycle. The second obstacle is the lack of bitwise shift operation in the proposed set of PE operations. This operation can be introduced into the set; however, at the date, the necessity for this operation in PEs is debatable. The wide input range of the SoftMax activation increases the complexity of its hardware implementation. To reduce the range, most of the mentioned papers suggest subtracting the maximum input value from all inputs [29; 31; 32; 34]. This is possible due to Equation (2). If \(c\) is equal to \(-max(x_{1},\ldots x_{n})\), the results of the exponential function are always between 0 and 1. However, the subtraction leads to negative powers and small values of the exponential function, resulting in low precision for fixed-point numbers. Since the proposed implementation of RCE processes 16-bit fixed-point numbers, this solution has a significant impact on the results obtained. In this paper, we have modified this approach to provide greater accuracy. \[S(x_{1},\ldots x_{N})_{i}=\frac{\exp(x_{i})}{\sum_{1}^{N}\exp(x_{n})}=\frac{ \exp(c)*\exp(x_{i})}{\exp(c)*\sum_{1}^{N}\exp(x_{n})}=\frac{\exp(x_{n}+c)}{ \sum_{1}^{N}\exp(x_{n}+c)} \tag{2}\]  
where \(c\) is any constant value. The hardware implementation of division is a well-recognized problem that increases the complexity of the design. A simplified implementation of this operation can be achieved using subtraction and bitwise shift, as shown in [29; 30; 33]; or by addition and bitwise shift, according to [34]. The problem of applying bitwise shift in RCE was discussed above.