# Feedback-Forward Neural Networks for Failure Mechanics Problems
## 2 Theory of Neural Networks
### Artificial Neural Network (ANN)  
ANN is defined as a mathematical model divided into a series of interconnected elements classified in layers, whose geometry and functionality have been compared with those of the human brain. ANN is made up of neurons that have one scalar output and multiple inputs, as sketched in Figure 1.  
An inventive but simple structure is composed of four parts: (i) input values, (ii) weights and bias, (iii) weighted total (sum), and (iv) activation function (threshold unit). A schematic diagram of an artificial neuron is illustrated in Figure 1, where, \(x_{1}-x_{3}\) are inputs, \(w_{1}-w_{3}\) are their corresponding weights, \(b\) is the bias, \(f\) is the activation function applied to the weighted sum of the inputs and \(\widehat{y}\) is the output of the neuron.  
Mathematically, this relation can be written as follows:  
\[\widehat{y}=f(\sum_{i=1}^{N}x_{i}w_{i}+b)\,, \tag{1}\]  
Figure 1: Schematic representation of an artificial neuron.  
where \(N\) is the number of samples. Equation (1) can be rewritten in the following compact form as follows:  
\[\widehat{y}=f(\mathbf{x}^{T}\mathbf{W}+b)\quad\text{with}\quad\mathbf{x}=[x_{1},\dots,x_{N} ]^{T}\quad\text{and}\quad\mathbf{W}=[w_{1},\dots,w_{N}]^{T}\;. \tag{2}\]  
The activation function \(f\) (also known as the transfer function) determines the output of neurons in the neural network model, its computational efficiency and its ability to train and converge after multiple iterations of training. This introduces non-linear properties to the network. Hence, its main purpose is to convert the input signal of a node in an ANN to an output signal. The output signal is then used as an input in the next layer of the neural network.  
Specifically, in ANN, a sum of products is performed for inputs \(\mathbf{x}\) and the corresponding weights \(\mathbf{W}\). Next, the activation function \(f(x)\) is applied to obtain the output of that layer and feed it as an input to the next layer. Activation functions can be linear and non-linear. A linear activation function (e.g., Purelin activation function) contains an infinite range and has no effect on the complexity of the data set. On the other hand, non-linear activation functions (e.g., Sigmoid and Tanh activation functions) introduce non-linearity in order to better learn the complex relationship between the input and output data. Here, the most widely used classic activation functions with their first derivatives are presented as follows:  
1. Sigmoid transfer function. \[f(x){=}\frac{1}{e^{-x}+1}\qquad;\qquad f^{\prime}(x){=}\frac{e^{-x}}{(e^{-x}+ 1)^{2}}\] (3)
2. Hyperbolic tangent (Tan-Sigmoid transfer function). \[f(x){=}\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}\qquad;\qquad f^{\prime}(x){=}1-\frac{ (e^{x}-e^{-x})^{2}}{(e^{-x}+e^{x})^{2}}=4\frac{e^{2x}}{(1+e^{2x})^{2}}\] (4)
3. Purelin function. \[f(x){=}x\qquad;\qquad f^{\prime}(x){=}1\] (5)  
Figure 2 illustrates those functions and their derivatives.  
A key aspect of the activation function is that it should be differentiable for performing a successful backpropagation optimization strategy. In this paper, we used the _hyperbolic tangent function_, due to its flexible range and general applicability in different engineering problems; see, for example, [53].