# The Limits of SEMA on Distinguishing Similar Activation Functions of Embedded Deep Neural Networks
## 2 Background
### Neural Network
#### 2.1.2 Activation Functions  
In [15; 16], the activation function set recovered included four activation functions, which are the Sigmoid function, Tanh function, softmax function, and rectified linear unit (ReLU) function. These functions were selected for the popularity of the functions. In this work, we also used the Leaky ReLU function [20], SELU function [17], Swish function [18], and Fast Sigmoid function [19].  
For the identification, the ReLU and softmax functions are very easy to be identified from others due to the distinct calculation complexity. The only challenge one has is to distinguish the Tanh and Sigmoid functions, as only these two have similar calculation complexities. However, there are many similar activation function proposed and used in AI. Using the SEMA attack proposed in previous work, we attempted to distinguish more activation functions. We chose four activation functions that have similar operations to the functions used in our previous work to examine the versatility of this attack in identifying the activation functions.  
A brief introduction of them is as follows:  
**Sigmoid**: The Sigmoid (logistic) function, i.e., Equation (2), is a nonlinear function that plots arbitrary inputs to outputs in the range \((0,1)\).  
\[h(a)=\frac{1}{1+e^{-a}} \tag{2}\]
**Tanh**: The Tanh function, i.e., Equation (3), is a rescaling of the Sigmoid function. The Tanh function maps arbitrary inputs to outputs in the range \((-1,1)\). Different from Sigmoid, Tanh is symmetric by the origin.  
\[h(a)=\frac{e^{a}-e^{-a}}{e^{a}+e^{-a}}=\frac{2}{1+e^{-2a}}-1 \tag{3}\]
**Softmax**: The softmax function, i.e., Equation (4), can map values into several outputs (or classes), the sum of which becomes 1. The output range is \((0,1)\). It is able to be seen as a probability and is used for classification problems.  
\[\begin{split} h(\mathbf{a})_{j}&=\frac{e^{a_{j}}}{ \sum_{k=1}^{K}e^{a_{k}}},\text{ for }j=1,\dots,K\\ \text{and }\mathbf{a}&=(a_{1},\dots,a_{K})\in \mathbb{R}^{K}\end{split} \tag{4}\]  
Figure 1: Multilayer Perceptron.  
**ReLU**: The ReLU function, i.e., Equation (5), is an extremely simple function, which can reduce the time of training and computing. It is mainly used as an activation function for deep neural networks (DNNs).  
\[h(a)=\begin{cases}0,&\text{for }a\leq 0\\ a,&\text{for }a>0\end{cases} \tag{5}\]
**Leaky ReLU**: The Leaky ReLU function, i.e., Equation (6), is similar to the ReLU function, except that it enables back propagation, even for negative input values by having the positive slope proposed in [20], where \(\beta=0.01\).  
\[h(a)=\begin{cases}\beta a,&\text{for }a\leq 0\\ a,&\text{for }a>0\end{cases} \tag{6}\]
**SELU**: The SELU function, i.e., Equation (7), is an improvised version of the ReLU function for self-normalizing neural networks, a type of feed-forward neural network [17]. The SELU function has parameters \(\lambda\) and \(\alpha\), which have flexible values according to the applications, while we fixed them as \(\lambda=1.0507\), \(\alpha=1.6733\).  
\[h(a)=\lambda\begin{cases}\alpha(\text{e}^{a}-1),&\text{for }a\leq 0\\ a,&\text{for }a>0\end{cases} \tag{7}\]
**Swish**: The Swish function, i.e., Equation (8), is a self-gated activation function proposed in [18]. Swish is extremely similar to the Sigmoid function, where the flexible \(\beta\) parameter is fixed as \(\beta=1\) in this work.  
\[h(a)=\frac{a}{1+\text{e}^{-\beta a}} \tag{8}\]
**Fast Sigmoid**: Some activation functions include the exponential function, which is expensive to compute. [PERSON] et al. evaluated several function approximation techniques to replace the original function with faster execution at the cost of some loss of accuracy [19]. We used one of them in our work, which we call the Fast Sigmoid function, i.e., Equation (9). This function is a cheap alternative to the original Sigmoid function with good enough accuracy. The flexible \(n\) parameter was fixed as \(n=1\) and \(n=9\) in this work.  
\[h(a)=\frac{1}{1+\text{e}^{-a}},\text{e}^{a}=(1+\frac{a}{2^{n}})^{2^{n}} \tag{9}\]