## 4 ML Instruction Set Architecture
### Activation Functions
The approximation of the _tanh_ function is much more complex and computationally intensive as it involves the computation of two exponential terms, posing exploding behavior for larger negative and positive x values, as shown in Figure 5. The exploding functional behavior is relaxed for the sigmoid function by computing the _sigmoid_ function only for positive x-values and using a logarithmic base function, finally mirroring and flipping the result for negative x-values, which does not prevent exploding gradients in the case of the _tanh_ function. _tanh_ can be rewritten as shown in the following Eq., computing the discretized _tanh_ using the same approach as used for the _sigmoid_ function:  
\[tanh(x)=sgn(x)\bigg{(}1-\frac{2}{e^{2|x|}+1}\bigg{)} \tag{3}\]  
The LUT table can be computed with a stretched x distribution as follows, assuming \(\Delta x=1,2,3,...\):  
\[log10 lut=\bigg{\{}int\bigg{(}log_{10}\bigg{(}\frac{i}{10}\bigg{)}100\bigg{)}:i \in\mathbb{I},10\leq i<100\wedge(i-10)\%\Delta x=0\bigg{\}} \tag{4}\]  
with % as the modulo operation that creates an equidistant series of values. The _log10 lut_ table size is \(90/\Delta x\) with an unsigned byte data type. The accuracy (relative error) of the sigmoid approximation is plotted in Figure 6 with an input and output scaling factor of 10 for different LUT sizes. The LUT sizes were 90, 45, and 23, respectively. Using \(\Delta x\) larger than 1 results in a significantly increased approximation error for small x-values (20%), but the average relative error rises only from 1% to 3%. Figure 5: Exploding output values for negative x-values (\(e^{-x}\) term) and positive x-values (\(e^{x}\) term) of the exponential function. ```
staticub1 log101 but[]={<90/DXvalues>} //x-scaleis1:10 andlog10-scaleis1:100
``` sb2 fplog10(sb2x){ sb2 shift=0; while(x>=100){shift++;x/=10;}; returnshift+100+(sb2)log101 but[x-10]; } staticub1 sglutA[]={<24 values>};//alt.ub2 staticub1 sglutB[]={<6 elements>}; //yscaleI:1000[0,1],xscale1:1000 ```
sb2 fpsigmoid(sb2x){ sb2y}{ ub1 mirror=x<0?1:0; if(mirror)==x; if(x>=RC1*1000)returnmirror?0:1000; if(x<=RA1*1000){ y=<00+((x*231)/1000); returnmirror?1000-y:y; }elseif(x<RB1*1000){ ub2i10=fplog10((x/5)/2)-OA1; y=((sb2)sglutA[i10]+YAl; returnmirror1000-y:y; } else{ ub2i10=fplog10((x/10)/10)-OB1; y=((sb2)sglutB[i10])+YB1; returnmirror1000-y:y; } return0; } staticub1 thutA[]={<24 values>};//alt.ub2 staticub1 thutB[]={<6 elements>};//yscale1:1000
``` //yscale1:1000[0,1],xscale1:1000 ```
sb2 fptanh(sb2x){ sb2y; ub1 mirror=x<0?1:0; if(mirror)==x; if(x>=RC2*1000)returnmirror?-1000:1000; if(x<=RA2*1000){ y=(x*920)/1000; returnmirror?-y:y; } elseif(x<RB2*1000){ ub2i10=fplog10((x/2)/2)-OA2 y=thutA[i10]+YAl2; returnmirror?-y:y; } else{ ub2i10=fplog10((x/10)/10)-OB2, y=thutB[i10]+YB2; returnmirror?-y:y; } return0; }
```

**Algorithm 1**Range-segmented and LUT-based implementation of the sigmoid and hyperbolic tangent functions with less than 1% approximation error for a wide range of x-values (using approximated LUT-based log10 function). Shown is the C program code. The data types are in the format: s = signed or u = unsigned, b = byte, and the number gives the number of bytes. The _fpsigmoid_ function LUTs are computed iteratively using the _fplog10_ function for LUT index stretching, described by the following pseudo-code in Algorithm 2. The symmetry of the sigmoid function is exploited by just computing the positive normalized x-range and applying mirroring and flipping for negative values. The positive function is approximated by four segments. The normalized x-range [0, 1] is handled by a linear function directly computable in the range [0, \(RA\)], followed by the first highly non-linearsegment in the range (\(RA\), \(RB\)), and the converging segment in the range (\(RB\), \(RC\)), and finally a constant segment (x \(\geq RC\)).