# Feedback-Forward Neural Networks for Failure Mechanics Problems
## 2 Theory of Neural Networks
### Feed-Forward Neural Network (FFNN)  
In a FFNN, the number of inputs and outputs are fixed and the knowledge of history variables is disregarded. According to the complexity of the training data, the architecture  
Figure 2: Transfer functions. **Left**: Sigmoid transfer function; **middle**: Hyperbolic-tangent transfer function; and **right**: Purelin transfer function.  
of the FFNN (to be more specific, the number of hidden layers and neurons in each layer) has to be determined. In this paper, Machine Learning Toolbox (nntraintool) is used to train the model under consideration (nntraintool is a Matlab toolbox that is used for training neural networks. It is divided into four parts, such as the neural network architecture, algorithms, training progress, and plots. Interested readers are referred to [Neural network training tool--MATLAB nntraintool (mathworks.com accessed on 7 July 2021)]).  
The formulation of a fully-connected feed forward neural network with two hidden layers \(\mathbf{H}\), shown in Figure 3, is defined as follows:  
\[\widehat{\mathbf{y}}=f(\mathbf{H}_{2}\mathbf{W}_{3}+\mathbf{b}_{3})\quad\text{with}\quad\mathbf{H} _{2}=f(\mathbf{H}_{1}\mathbf{W}_{2}+\mathbf{b}_{2})\quad\text{and}\quad\mathbf{H}_{1}=f(\mathbf{x} \mathbf{W}_{1}+\mathbf{b}_{1})\, \tag{6}\]  
where the vector \(\widehat{\mathbf{y}}\) is the output, the input vector \(\mathbf{x}\) contains the features of a sample, \(\mathbf{W}\) is the weight matrix, and \(\mathbf{b}\) is the bias vector for each respective layer. In the current study, hidden layers have the tangent hyperbolic activation function \(f\), which is formulated as follows:  
\[tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}. \tag{7}\]  
This function is a rescaling of the logistic Sigmoid function, depicted in Figure 2, with an output range of \([-1,1]\). The hidden layers values are included in the vector \(\mathbf{H}\). The feed forward neural network architecture can be simplified as follows, given that \(\mathbf{y}\) is the true function:  
\[\widehat{\mathbf{y}}=\widehat{\mathbf{y}}(\mathbf{x},\mathbf{W},\mathbf{b})\quad\text{and}\quad \big{\{}\mathbf{b}^{\prime},\mathbf{W}^{\prime}\big{\}}=\text{Arg}\ \Big{\{}\min_{\mathbf{b},\mathbf{W}}\ \mathcal{L}\big{(}\widehat{\mathbf{y}},\mathbf{y}\big{)}\Big{\}}\, \tag{8}\]  
where the loss function \(\mathcal{L}\) is minimized to find the optimized weights \(\mathbf{W}^{\prime}\) and biases \(\mathbf{b}^{\prime}\) of the trained neural network model. In this analysis, a feed forward neural network with two hidden layers (we have tried different architectures with hidden layers of different sizes; however, the final result was not affected. Therefore, for simplicity, we have kept the neural network architecture with two hidden layers throughout this research) of 10 neurons each is created such that _strains_\(\mathbf{\varepsilon}\) are features and _stresses_\(\mathbf{\sigma}\) are the outputs of the model.  
Mean-squared error (MSE) is considered the loss function \(\mathcal{L}\) such that it is minimized during the training process as follows:  
\[MSE=\frac{1}{N}\sum_{i=1}^{N}\!\left[(x_{target})_{i}-(x_{pred})_{i}\right]^{2}. \tag{9}\]  
Figure 3: A fully-connected feed-forward neural network. The input layer \(\mathbf{x}\) has three features, each of the two hidden layers has 10 neurons and the output layer \(\widehat{\mathbf{y}}\) has three outputs.  
Since the stress and strain values are on different scales of magnitude, it is important to scale the data set to a comparable range; therefore, the input and target data are initially scaled to be in the range \([-1,1]\) by using the following formula:  
\[\tilde{x}=\frac{2(x-x_{min})}{x_{max}-x_{min}}+1\;, \tag{10}\]  
where \(\tilde{x}\) is the scaled or normalized value of \(x\) in the range of \([-1,1]\).