# Implementation of the SoftMax Activation for Reconfigurable Neural Network Hardware Accelerators
## 4 Implementation of the SoftMax Activation in RCE
### Reduction of the Range Width of Input Values  
As mentioned in the previous subsection, the proposed implementation of the SoftMax activation supports inputs in the range from \(-2.5\) to \(2.5\). This range was chosen for two reasons--it is centered around zero, and its width is equal to five. According to Equation (3),  
\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Step** & **Range** & **a** & **b** & **Masked Point** \\ \hline
1 & \([-2.5,-2.0)\) & 0.1015625 & 0.3359375 & 1111 1101 111111 \\
2 & \([-2.0,-1.5)\) & 0.1796875 & 0.4921875 & 1111 1110 011111 \\
3 & \([-1.5,-1.0)\) & 0.2890625 & 0.65625 & 1111 1110 11111 \\
4 & \([-1.0,-0.5)\) & 0.4765625 & 0.84375 & 1111 1111 01111 \\
5 & \([-0.5,0.0)\) & 0.7890625 & 1 & 1111 1111 11111 \\
6 & \([0.0,0.5)\) & 1.296875 & 1 & 0000 000 0111111 \\
7 & \([0.5,1.0)\) & 2.1328125 & 0.58203125 & 0000 000 111111 \\
8 & \([1.0,1.5)\) & 3.53125 & \(-0.81640625\) & 0000 0001 0111111 \\
9 & \([1.5,1.75)\) & 5.09375 & \(-3.16015625\) & 0000 001 10111111 \\
10 & \([1.75,2.0)\) & 6.53125 & \(-5.67578125\) & 0000 001 11111111 \\
11 & \([2.0,2.25)\) & 8.390625 & \(-9.39453125\) & 0000 0010 0011 11111 \\
12 & \([2.25,2.5]\) & 10.78125 & \(-14.7734375\) & 0000 010 01111111 \\ \hline \hline \end{tabular} The values of the applied masks are underlined.  
\end{table}
Table 1: The parameters of the exponential function approximation.  
Figure 9: Implementation of the exponential function in the RCE.  
the result of SoftMax activation is a ratio between the results of the exponential function. The reduction of the input range leads to the exclusion of some values, introducing an error in the results. To make this effect less considerable, the range should be wide enough to exclude the least significant values only. Insofar as the result of SoftMax activation is probability distribution, the introduced error can be defined as the ratio between the exponential function at the maximum point and the exponential function at the minimum point:  
\[s=\exp(x_{l})/\exp(x_{h}) \tag{4}\]  
where \(x_{l}\) and \(x_{r}\)--the lowest and highest points of the chosen range, respectively.  
In this paper, the error is assumed equal to 0.0075, so the difference between the highest and the lowest possible inputs equals to:  
\[x_{h}-x_{l}=-ln(0.0075)=4.89\approx 5 \tag{5}\]  
To avoid the range of rapid growth of the exponential function, as well as processing small numbers in the negative subrange, it is proposed to select the approximation range endpoints equidistant from zero. Therefore, in this paper, the exponential function is approximated in the range from \(-2.5\) to \(2.5\).  
Obviously, if some of the input values exceed the upper endpoint, omitting those values will introduce a significant error to the result. As mentioned previously, a special shift can be applied to all input values to address this issue (Equation (2)). The shift value can be chosen as the difference between the upper endpoint of the approximation range and the maximum input value:  
\[c(x_{1},\ldots x_{N})=x_{h}-max(x_{1},\ldots x_{N})=2.5-max(x_{1},\ldots x_{N}) \tag{6}\]  
After applying this shift, all the inputs will be limited to the upper endpoint of the approximation range.