## 3 Methodology
### Hybrid Activation Function  
The activation function is used to determine the activation and deactivation of output data when a signal of input data is received in the neural network structure. It is important to select an activation function that is suitable for the purpose of the study because the results are dependent on which activation function is used. In general, activation functions include linear and nonlinear functions. The linear function changes by a constant multiple of the input and the form of the linear function is a straight line. However, the use of the linear function as an activation function is not an advantage of the neural network. On the other hand, the non-linear functions have two or more straight or curved forms and typical nonlinear functions used in neural network are the sigmoid function, the hyperbolic tangent function, the rectified linear unit (ReLU) function, etc. The sigmoid function has the feature of a fast learning speed. In addition, it has a value of 0 to 1 in the range of \(-\infty\) to \(\infty\). The form of the sigmoid function is shown in Equation (7):  
\[\sigma(x)=\frac{1}{1+e^{-x}} \tag{7}\]  
\[\sigma^{\prime}(x)=\sigma(x)(1-\sigma(x)) \tag{8}\]  
However, \(x\) has a value close to 0 in the range \(-\infty\) to \(\infty\) in the derivative form of the sigmoid function and it causes vanishing gradient problems in error back propagation [29].  
The hyperbolic tangent function can be expressed as the ratio of the hyperbolic sine function and the hyperbolic cosine function or as the ratio of the sum and difference of the two exponential functions. The form of the hyperbolic tangent function is shown in Equation (9):  
\[tanh(x)=\frac{sinh(x)}{cosh(x)}=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}} \tag{9}\]  
\[tanh^{\prime}(x)=1-tanh^{2}(x) \tag{10}\]  
The hyperbolic tangent function is like the sigmoid function, it has a value of \(-1\) to \(1\) in the range \(-\infty\) to \(\infty\). However, the hyperbolic tangent function also has a value close to 0 when \(x\) is \(-\infty\) and \(\infty\) in the derivative form which causes a problem of vanishing gradient during back propagation [30].  
The rectified linear unit (ReLU) function was first introduced by [PERSON] et al. [28] and was used as an activation function in neural network model. The ReLU function changes its form of function, depending on the range of \(x\). The ReLU function is shown in Equation (11):  
\[f(x)=\left\{\begin{array}{lcl}0&&\quad\text{if}&x<0\\ x&&\quad\text{if}&x\geq 0\end{array}\right. \tag{11}\]  
\[f^{\prime}(x)=\left\{\begin{array}{lcl}0&&\quad\text{if}&x<0\\ 1&&\quad\text{if}&x\geq 0\end{array}\right. \tag{12}\]  
The ReLU has a value of 0 when \(x\) is less than 0, and \(f(x)=x\) when \(x\) is greater than 0, as shown from the above function equation. However, the ReLU function has a dying neuron problem because \(f(x)\) always has a value of zero when the value of \(x\) is less than zero. Moreover, it has the limitation of overfitting the results [30].  
In this study, the hybrid activation function was proposed to solve the problems of general activation functions such as the vanishing gradient problem, the dying neuron problem, etc. The hybrid activation function is composed of a hyperbolic tangent function and a rectified linear unit function. The form of the hybrid activation function is as follows:  
\[f(x)=w_{1}ReLU(x)+w_{2}tanh(x) \tag{13}\]  
\[f^{\prime}(x)=\left\{\begin{array}{lcl}w_{1}+w_{2}\big{(}1-tanh^{2}(x)\big{)}& \qquad\text{if}&x\ <0\\ w_{2}\big{(}1-tanh^{2}(x)\big{)}&\qquad\text{if}&x\ \geq 0\end{array}\right. \tag{14}\]  
where \(w_{1}\) and \(w_{2}\) are weights, and each sum of weights is one (\(\sum\limits_{i}^{n}w_{i}=1.0\)). As the weights (\(w_{1}\) and \(w_{2}\)) change, the range of the hybrid activation function occupies the hatched part of the figure (see Figure 1).