## 2 Artificial Neural Network Flow Law
### Artificial Neural Network Equations
#### 2.1.2 Activation Functions  
At the heart of ANNs lies the concept of activation functions, pivotal elements that determine how information is transformed within the neurons. Choosing activation functions is a critical design decision, as these functions greatly influence the network's capacity to learn and represent complex patterns in data. The selection of activation functions is guided by their distinct properties, including non-linearity, differentiability, and computational efficiency. In regression ANNs, the choice of activation functions is typically driven by the need to approximate continuous output values rather than class labels. Many studies have been proposed concerning the right activation function to use depending on the physical problem to solve, such as the reviews proposed by [PERSON] et al. (2018) or [PERSON] et al. (2019). The activation function is essential for introducing non-linearity into a neural network,  
Figure 2: Two hidden layers ANN architecture with 3 inputs (\(\varepsilon\), \(\hat{\varepsilon}\) and \(T\)) and 1 output (\(\sigma\)). allowing it to capture non-linear features. Without this non-linearity, the network would behave like a linear regression model, as emphasized by [PERSON] et al. (2018). A number of activation functions can be used in neural networks. In our previous published work ([PERSON] et al., 2018; [PERSON] et al., 2018), we have mostly used the Sigmoid activation function for the ANN flow laws. In the present paper, we are going to explore other activation functions and their influence on the final results, up-to the implementation into a FE software. Among the number of activation functions available in the literature, we have selected the six ones reported in Figure 3.  
The Sigmoid activation function (Sigmoid, 2000), also known as the logistic activation function, is widely used in ANNs. It was originally developed in the field of logistic regression and later adapted for use in neural networks. It maps any input to an output in the range \([0,1]\), making it suitable for tasks where the network's output needs to represent probabilities or values between 0 and 1. The Sigmoid activation function \(f(x)\) and its derivative \(f^{\prime}(x)\) are defined by:  
\[f(x)=\frac{1}{1+\mathrm{e}^{-x}}\qquad\text{and}\qquad f^{\prime}(x)=f(x)\cdot [1-f(x)]. \tag{5}\]  
This function has been widely used until the early 1990s. Its main advantage is that it is bounded, while its main drawbacks are the problem of vanishing gradient, a non-centered on zero output and saturation for large input values. From the 1990s to 2000s, the hyperbolic tangent activation function has been introduced and was preferred to the Sigmoid function for the training of ANNs. The hyperbolic tangent function squashes the output within the range \([-1,+1]\), and its formulation is given by the following equations:  
\[f(x)=\frac{\mathrm{e}^{x}-\mathrm{e}^{-x}}{\mathrm{e}^{x}+\mathrm{e}^{-x}} \qquad\text{and}\qquad f^{\prime}(x)=1-f(x)^{2}. \tag{6}\]  
This function is useful when the network needs to model data with mean-centered features, as it can capture both positive and negative correlations. The Tanh activation function and the Sigmoid activation function are closely related in the sense that they both introduce non-linearity and squash their inputs into bounded ranges. The evaluation of this activation function requires more CPU time than the Sigmoid function, since we need to compute two exponential functions (\(\mathrm{e}^{x}\) and \(\mathrm{e}^{-x}\)) to evaluate \(f(x)\). ReLU is a classic function in classification ANNs due to its simplicity and computational efficiency, as it involves simple thresholding. It introduces non-linearity and is computationally efficient. It outputs the input if it is positive and zero if it is negative:  
\[f(x)=\max(0,x)\qquad\text{and}\qquad f^{\prime}(x)= \begin{cases}1&x>0\\ 0&x\leq 0\end{cases} . \tag{7}\]  
Figure 3: Activation functions used in ANNs.