# Hands-On Fundamentals of 1D Convolutional Neural Networks--A Tutorial for Beginner Users
## 6 Other Activation and Loss Functions  
In addition to the sigmoid as an activation function, which was considered in Section 5, here, we aim to extend the calculation to also consider the hyperbolic tangent (tanh) [13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 82; 84; 86; 87; 88; 89; 91; 85; 89; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 111; 111; 112; 113; 1 logarithmic hyperbolic cosine (\(\log\cosh\)) [135], and Rectified Linear Unit (ReLU) [136], including some of its variants [129; 137; 138]. Moreover, along with MSE, other loss functions are explored for both regression and classification problems. To provide the correct expressions for the different cases, in backpropagation, what needs to be modified are only the expressions for \(\Delta y_{out}(i)\), \(\Delta C_{l}^{(2)}(i)\), and \(\Delta C_{j}^{(1)}(i)\). Once computed, they can be used in Equations (15)-(33). Generally, the choice of the activation function depends on which layer you are dealing with [139]. Although ReLU is commonly preferred in hidden layers (here \(CL_{1}\) and \(CL_{2}\)), there are examples of its use in output layers [140] when the response variable is continuous and greater than zero [141]. In this section, we assume that the ReLU activation function is applied exclusively in hidden layers, while the other activation functions are used in both the output and hidden layers. Let us start by discussing the regression task. The various configurations explored in terms of activation and loss functions for this task are summarized in the Tables 2 and 3. The expression for \(\Delta y_{out}(i)\) depends on both the loss and the activation functions considered in the output layer. When MSE is used with the sigmoid, the valid expression is the one provided in Section 5. Instead, when \(\tanh\) is used, we have  
\[\Delta y_{out}(i)=(y_{out}(i)-y_{t}(i))\cdot\Big{(}1-y_{out}^{2}(i)\Big{)}. \tag{34}\]  
where the first factor comes from the derivative of MSE, and the second one from the derivative of \(\tanh\), which is \(1-\tanh^{2}\). Another loss function commonly used in regression problems is represented by the \(\log\cosh\) function  
\[L=\frac{1}{N}\sum_{i=1}^{N}(\log(\cosh(y_{out}(i)-y_{t}(i)))) \tag{35}\]  
where \(i\) runs from \(1\) to the dimension of the output (here, \(N=2\)). With \(\log\cosh\) as loss function, the expression (34) becomes  
\[\Delta y_{out}(i)=\frac{1}{2}\tanh(y_{out}(i)-y_{t}(i))y_{out}(i)\cdot(1-y_{out }(i)) \tag{36}\]  
for sigmoid activation function in the output layer, and  
\[\Delta y_{out}(i)=\frac{1}{2}\tanh(y_{out}(i)-y_{t}(i))\cdot\Big{(}1-y_{out}^{ 2}(i)\Big{)} \tag{37}\]  
for \(\tanh\) activation in the output layer. As in Equations (34)-(36), the first factor comes from the derivative of the loss function, and the second comes from that of the activation function. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Loss Function** & **Activation Function in Output Layer** & \(\Delta y_{out}\) \\ \hline MSE & sigmoid & (15) \\ MSE & \(\tanh\) & (34) \\ \(\log\cosh\) & sigmoid & (36) \\ \(\log\cosh\) & \(\tanh\) & (37) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Expressions for \(\Delta y_{out}\) corresponding to different loss and activation functions in the output layer. The different activation functions available for hidden layers have an impact on both \(\Delta C_{l}^{(2)}(i)\) and \(\Delta C_{j}^{(1)}(i)\) expressions. When the sigmoid is used, the valid expressions are those provided in Section 5. Instead, when the \(\tanh\) activation function is used, we have  
\[\Delta C_{l}^{(n)}(i)=\Delta C_{l,\sigma}^{(n)}(i)\left(1-\left(C_{l,\sigma}^{ (n)}(i)\right)^{2}\right) \tag{38}\]  
where \(n=1,2\) represents the hidden layer number.