# Hands-On Fundamentals of 1D Convolutional Neural Networks--A Tutorial for Beginner Users
## 6 Other Activation and Loss Functions
Moreover, with the ReLU activation function in hidden layers defined as follows:  
\[ReLU(x)= \begin{cases}x&x\geq 0\\ 0&x<0\end{cases} , \tag{39}\]  
Equation (38) can be rewritten:  
\[\Delta C_{l}^{(n)}(i)=\Delta C_{l,\sigma}^{(n)}(i) \begin{cases}1&C_{j}^{(n)}( i)\geqslant 0\\ 0&C_{l}^{(n)}(i)<0\end{cases}  \tag{40}\]  
where \(n=1,2\) represents the hidden layer number. Among the possible variants of teh ReLU activation function, we can consider ELU (Exponential Linear Unit):  
\[ELU(x)= \begin{cases}x&x>0\\ \alpha(\varepsilon^{x}-1)&x\leq 0\end{cases}  \tag{41}\]  
and Leaky ReLU  
\[LeakyReLU(x)= \begin{cases}x&x\geq 0\\ \alpha x&\leq 0\end{cases}  \tag{42}\]  
with \(\alpha\) a parameter that might be considered as trainable. The expression (38) is now  
\[\Delta C_{l}^{(n)}(i)=\Delta C_{l,\sigma}^{(n)}(i) \begin{cases}1&C_{l}^{(n)}(i )>0\\ \alpha e_{l}^{(n)}(i)&C_{l}^{(n)}(i)\leq 0\end{cases}  \tag{43}\]  
for ELU, and  
\[\Delta C_{l}^{(n)}(i)=\Delta C_{l,\sigma}^{(n)}(i) \begin{cases}1&C_{l}^{(n)}( i)\geqslant 0\\ \alpha&C_{l}^{(n)}(i)<0\end{cases}  \tag{44}\]  
for Leaky ReLU. Let us shift our focus to classification tasks. In these scenarios, the output layer provides probabilities for decision-making when classifying input data. Essentially, the output is presented as a list of probabilities for different potential labels. When calculating the error, the target output \(y_{t}(i)\) is represented by a one-hot vector that has 1 at one index \begin{table}
\begin{tabular}{c c} \hline \hline
**Activation Function in Hidden Layer** & \(\Delta C^{(n)}\) \\ \hline sigmoid & (20),(30) \\ tanh & (38) \\ ReLU & (40) \\ ELU & (43) \\ Leaky ReLU & (44) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Expressions for \(\Delta C^{(n)}\) corresponding to different activation functions in hidden layers. and 0 at all other indices. Consequently, this implies that \(\sum_{k=1}^{N}y_{t}(k)=1\). Since the output of the output layer may assume both positive and negative real values, the output \(y_{out}(i)\) needs to be transformed into probability-like values. This can be achieved using the Softmax function:  
\[P_{i}=\frac{e^{y_{out}(i)}}{\sum_{k=1}^{N}e^{y_{out}(k)}} \tag{45}\]  
where \(0\leq P_{i}\leq 1\) and \(\sum_{i=1}^{N}P_{i}=1\). The derivative of Equation (43) can be easily calculated as follows:  
\[\frac{\partial P_{k}}{\partial y_{out}(i)}= \begin{cases}-P_{k}P_{i}&k\
eq i\\ P_{k}(1-P_{i})&k=i.\end{cases}  \tag{46}\]  
When using the Softmax as the output activation function, the loss function that is generally associated is represented by the cross-entropy, given by  
\[L=-\sum_{i=1}^{N}y_{t}(i)\ln(P_{i}). \tag{47}\]  
The expression for \(\Delta y_{out}(i)\) can be derived after calculating the derivative of the loss, which is obtained using the derivative of Equation (44):  
\[ \begin{split}\frac{\partial L}{\partial y_{out}(i)}& =\sum_{k=1}^{N}\frac{\partial L}{\partial P_{k}}\frac{\partial P _{k}}{\partial y_{out}(i)}\\ &=\sum_{k=1}^{N}\left(\frac{y_{t}(k)}{P_{k}}P_{i}\right)-\frac{y_ {t}(i)}{P_{i}}P_{i}\\ &=P_{i}-y_{t}(i).\end{split}  \tag{48}\]