## 4 ML Instruction Set Architecture
### Activation Functions  
Besides the vector operations that can be simply implemented in closed form with integer arithmetic, transfer functions with non-linear behavior are the most critical part of the integer computation of ML models. There are different transfers (activation) that are used in ANN and CNN models; the most prominent examples are:  
* Linear function (_linear_) without x- and y-limits;
* Logistic or sigmoid function (_sigmoid_) with y-limit = [-1, 1];
* Hyperbolic tangent function (_tanh_) with y-limit = [-1, 1];
* Rectifying linear unit (_relu_) with one-side open y-limit = [0, \(\infty\)). \[ \begin{array}{l}sigmoid(x)=\frac{1}{1+e^{-x}}\\ tanh(x)=\frac{e^{x}-e^{x}}{e^{x}+e^{-x}}\end{array}  \tag{2}\]  
The _linear_ and _relu_ functions can be directly implemented with integer arithmetic without loss of accuracy (except due to the integer discretization). The highly non-linear _sigmoid_ and _tanh_ functions require an approximation by using a hybrid approach combining a (compact) look-up table (LUT) and an interpolation function. The _tanh_ function can be neglected since it can be replaced, in most cases, by the _sigmoid_ function without loss of generalization (of course, prior to training). Trigonometric functions and functions composed of trigonometric functions are implemented with piecewise linear and non-linear look-up tables. The approximated discretized _sigmoid_ function algorithm is shown in Algorithm 1. For example, the error of the discretized sigmoid function is always less than 1% or below 10 digital values while only \begin{table}
\begin{tabular}{l} \hline \hline
**Vector Operation** \\ \hline vecfold \\ ( invec wgtvec outvecs scalevec -- ) \\ Performs a folding operation _ivec x upgtvec_ with a given filter. The weights vector _wgtvec_ must have the size \(\mathit{l}\mathit{l}\mathit{invec}^{\mathit{1}\mathit{*}}\mathit{outvec}\). \\ \hline vecconv \\ ( invec wgtvec outvecs scale inwidth wgtwidth stride pad -- ) \\ Performs a two-dimensional kernel-based convolution operation _ivec \(\bigotimes\) upgtvec_. The width of the input and kernel matrix (still a linear array) must be provided, and the width of the output and the heights are computed automatically from the vector lengths. If _wgtwidth_ is negative, a pooling operation is performed. The _wgtvec_ argument provides then the height of the filter and the operation to be performed. \\ \hline vecconv \\ ( invec wgtvec outvecs scale inwidth wgtwidth stride pad -- ) \\ Performs a two-dimensional pooling if _wgtwidth_ is negative. The _wgtvec_ argument provides the height of the filter and the operation to be performed (function index). \\ \hline vecmap \\ ( srcvec dstvec func scalevec -- ) \\ Maps all elements from the source array onto the destination array using an external (IOS) or internal (user-defined word) \\ function, e.g., the sigmoid function. \\ \hline vecvd \\ ( vec vecoff veclen op -- index value / valueL valueM ) \\ Reduces a vector to a scalar value. Supported operations are _min_ (1) and _max_ (2), returning position and value, _mean_ (4) and _average_ (8), returning a double-word value. \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Part 2 of the basic vector ANN functions operating on embedded or external array data (e.g., the sample buffer). requiring 30 bytes of LUT space and less than 10 unit operations (in addition to the LUT size of the _fplog10_ function). These software functions can be immediately implemented in hardware, too. The LUTs are computed with Algorithm 2.