# The Limits of SEMA on Distinguishing Similar Activation Functions of Embedded Deep Neural Networks
## 5 Identification Using SEMA
### Sigmoid, Tanh, and Swish  
Figure 4 compares the leakage patterns of the Sigmoid, Tanh, and Swish functions with positive inputs, in this case 4.154561. It can be observed that the Sigmoid and Swish functions are identical; however, the Tanh function has a slight difference in peaks. Although there is no obvious gap in the processing time, the peak patterns differ, and extra peaks can be seen in two sections from the Tanh function when comparing with the Sigmoid and Swish function.  
The extra peaks observed from the Tanh function are surrounded in a red box shown in Figure 4b. The first peak can be seen right after the multiplication. The second peak can be seen in the latter half of the activation function. The differences in peaks comes from the difference in the functions. The Tanh function has an additional multiplication and subtraction compared with the Sigmoid function. We observed that these operations cause the difference in the peak patterns. The Sigmoid function and Tanh function can be distinguished with the different peak patterns, with the Tanh function having more peak patterns.  
The processed leakage patterns of the Sigmoid and Swish functions with input 0 are shown in Figure 5. When processing input 0, the Swish function has 0 as the numerator, reducing the time for the division operation compared to the Sigmoid function. The division operation for the Sigmoid function is surrounded in a red box, and the division operation for the Swish function is surrounded in a green box. It can be clearly observed that the Swish function is shorter in computation time and also has fewer peaks compared to the Sigmoid function.  
The identification of activation functions by SEMA is mainly performed by observing the differences in peak patterns and the number of peaks. The differences in the shapes of the traces and peaks originates from the variability in the operations of the functions. The operations and peaks of the traces have a relation, allowing SEMA to distinguish the activation functions. The differences in just the value, however, do not cause changes to the peak patterns and number of peaks. The only difference between the Swish and  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Activation Function** & **Calculation Time** &
\begin{tabular}{c} **Approximate Range of** \\ **Computation Time (in \(\mu\)s)** \\ \end{tabular} \\ \hline ReLU, Leaky ReLU, SELU & Short & 5\(\sim\)15 \\ Sigmoid, Tanh, Swish & Medium & 80\(\sim\)210 \\ Softmax, Fast Sigmoid & Variable & 50\(\sim\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Classification of Similar Activation Functions Using Calculation Time.  
Sigmoid function is just the value of the numerator, which resulted in the same EM trace peak patterns. The same could be said for the peaks for the ReLU function with positive and negative inputs. We believe that the limit of SEMA is that only the operations become visible from the EM trace. Therefore, it is important to observe the timing for when the numerator is 0, because the computation will become much faster.  
\(M_{i}\): \(x_{i}\times w_{i}\) (Weight multiplication of the \(i^{\text{th}}\) node)  
\(A_{n}\): \(\sum_{i=1}^{n}(x_{i}\times w_{i})\) (Sum of the \(n^{\text{th}}\) node)  
\(b\): Bias  
\(h(a)=\dfrac{1}{1+e^{-a}}\)  
\(h(a)=\dfrac{\text{Tanh}}{1+e^{-2a}}-1\)  
\(h(a)=\dfrac{\text{Swish}}{1+e^{-a}}\)  
\(h(a)=\dfrac{a}{1+e^{-a}}\)  
Figure 4: Comparison of the Patterns of Sigmoid, Tanh, and Swish Functions for Positive Input. (**a**) Sigmoid Function. (**b**) Tanh Function. (**c**) Swish Function.