## 4 ML Instruction Set Architecture
### Activation Functions
```
RA :=1 RB:=3 RC:=108=0.01 OA :=int(fplog10(int(x*1000/5))/2) OB :=int(fplog10(int(x*1000/10))/10) YA :=int(F(RA)*1000) YB :=int(F(RB)*1000) luta :=[] forx=RAtoRBstep8 do i10 :=int(fplog10(int(x*1000/5))/2)-OA ifluta[i10] = undefinedthen luta[i10] :=int(F(x)*1000)-YA endif done luts :=[] forx=RBtoRCstep8 do i10 :=int(fplog10(int(x*1000/10))/10)-OB iflutB[i10] = undefinedthen lutB[i10] :=int(F(x)*1000)-YB endif done
```

**Algorithm 2** Computation of the segmented LUTs A/B for the integer-scaled sigmoid and hyperbolic tangent functions. F is the real-valued generator function for sigmoid or tanh LUT computation. The log10 function is used to stretch the x distribution in the LUTs. Figure 6: Relative discretization error of integer-scaled LUT-based approximation of the _log10_ function for different \(\Delta\)x values (1,2,4) and LUT sizes of 90, 45, and 23, respectively. The accuracy (relative error) of the sigmoid approximation is plotted in Figure 7 with an input and output scaling factor of 10,000 (i.e., 1:10,000). For \(x>-3\), the error is below 5% and decreases to 1% on average. For \(x<-3\), the relative error increases significantly due to the integer discretization error. The error increases in some x-ranges for lower _fplog10_ resolution (smaller LUT sizes, \(\Delta x=4\)) but can be improved if the sigmoid interval ranges \(R\) are shifted towards larger values (increasing the sigmoid LUT, too). The red and green areas show lowered or increased accuracy. The accuracy of the transfer function itself is not a measure of the accuracy of ML models using this function, especially if post-trained (adapted) using the discretized function instead of the continuous function. For the segment ranges \(R=\left\{A=1,\,B=3,\,C=10\right\}\), the sizes of the LUTs are \(\left\lvert\,sglutA\,\right\rvert=24\) and \(\left\lvert\,sglutB\,\right\rvert=6\), for \(R=\left\{1,\,7,\,15\right\}\), the sizes are \(\left\lvert\,sglutA\,\right\rvert=43\) and \(\left\lvert\,sglutB\,\right\rvert=8\), approximately \(\left\lvert\,sglutA\,\right\rvert\approx 6(B-A+1)\) and \(\left\lvert\,sglutB\,\right\rvert\approx C-B\), in addition to the LUT size of the _fplog10_ function (\(90/\Delta x\)), requiring in total \(6(B-A+1)+C-B+90/\Delta x\) Bytes of static storage. The implementation of the integer version of the _tanh_ function requires extended LUTs due to the higher gradient in the x-range [0, 1], i.e., choosing \(RA<1\). Typical results compared with the real-valued function are shown in Figure 8. Selected error statistics of the _fpsigmoid_ and _fptanh_ functions are shown in Table 5. The median error is mostly below 1%. Higher errors commonly occur with small y values as a result of integer discretization

Figure 7: Relative discretization error of integer-scaled LUT-interpolated approximation of the _sigmoid_ function using the discretized _log10_ LUT-interpolation function for different LUT resolutions and sigmoid segment ranges \(R\). The small error plots show only positive x values. and not the approximation itself. The LUT sizes vary between 20 and 50 elements and are small enough to be stored in very low-resource micro-controllers, even if a 16-bit data word size is required. To summarize, the accurate approximation of the highly non-linear and widely used _sigmoid_ and _tanh_ functions is possible with a segmented LUT approach. The computational complexity is low (less than 20 unit operations are required for one function evaluation), and the storage requirement is low (about 200 Bytes for each function). The average relative error is below 1%, except for small integer values limited by the discretization error. ## 5 Transformation Process

In the following, we introduce the transformation process of a floating-point arithmetic model to a scaled fixed-point (FP) integer arithmetic model. For historical reasons, we call the floating-point model Foo and the fixed-point integer-scaled model FooFP, and the same is true for the activation function, Act and ActFP, respectively.