# Implementation of the SoftMax Activation for Reconfigurable Neural Network Hardware Accelerators
## 4 Implementation of the SoftMax Activation in RCE
### Implementation of the Exponential Function  
As discussed earlier, in RCE, complex functions need to be decomposed among a group of processing elements; thus, each element performs a simple operation. One of the most popular approaches to defining a function in a simple general way is piecewise linear approximation (PLA). This approach can be used to implement the exponential function. We briefly discuss the implementation of PLA in the RCE using the example of sigmoid activation described in detail in the paper [25]. This implementation is based on the approximation with equal subranges--the entire approximation range is divided into several subranges of equal width. Then, in each subrange, the value of the original function is replaced by the straight line defined by the Equation (3). The \(a\) and \(b\) values are chosen to reduce the difference between the original and approximated functions in this subrange. The piecewise linear approximation is a popular approach to implement complex functions in hardware accelerators [25; 35; 36]. \[f(x)=ax+b \tag{3}\]  
where \(a\) and \(b\) are the multiplier and addend of the subrange in which the \(x\) is located. Due to the shape of sigmoid activation (Figure 7), the approximation in the range \([-5,5]\) with 10 subranges provides acceptable results [25; 35]. The input values below \(-5\) lead to zero output; the values above 5 result in one. Since all the subranges are calculated independently of each other, they can be processed simultaneously by different processing elements (Figure 8). Thus, each subrange is implemented in the form of a chain of PEs. Each chain has the GAT element that passes the computed value if the input value lies within the subrange of this chain. Otherwise, it outputs zero. To ensure whether the input value belongs to a certain subrange, it is necessary to compare this value with the reference value of the subrange, ignoring insignificant bits. The insignificant bits are the least significant bits that cannot affect the subrange of the signal. This can be achieved by applying a special mask to the input value. In this paper, it is proposed to use a mask containing ones in insignificant bits and zeroes in the rest of the bits. The application of this mask implies performing the bitwise OR operation on the mask and the input signal. Since the proposed models work with 16-bit fixed-point numbers with eight fractional bits, as well as approximation points are integer values (\(-5\), \(-4\), \(\ldots 4\), \(5\)), the eight least significant bits (fractional part) of the mask are equal to one, the rest are equal to zero. After applying this mask, the input value keeps its integer part unmodified, and its fractional part is always \"11111111\". Similarly, the subrange reference value is the  
Figure 7: Sigmoid activation. result of masking any point of the subrange. Thus, the GAT elements compare the masked input values with masked approximation points. Obviously, this approach can be applied to other functions as well. It is possible to use any approximation step with a length equal to a power of two (2, 1, 1/2, 1/4, etc.) by varying the applied mask. An approximation with any required number of subranges can be accomplished by altering the number of the involved PE rows. Nevertheless, as noted earlier, the exponential function is non-linear and fast-growing (Figure 6). As a result, the piecewise linear approximation of this function with equal steps introduces a significant error. One possible solution to this problem is an approximation with a variable step. In this paper, an approximation of the exponential function in the range [\(-\)2.5, 2.5] with 12 varying steps is proposed. The reasons for choosing such an interval will be discussed in the next subsection. The lengths of these steps are distributed as follows: from \(-\)2.5 to 1.5, the step is 0.5, and from 1.5 to 2.5, the step is 0.25. Both step lengths are a power of two, so they can be implemented using the masking operation discussed above.