# Efficient Deep Learning for Gradient-Enhanced Stress Dependent Damage Model
## 3 Training Data  
### Optimizers  
In machine learning (ML) approaches, the weights and biases of the network (see Figure 1) are obtained through minimizing an objective function. Commonly, gradient descent methods are employed in ML. The update of the gradient descent has the form \(x_{k+1}=x_{k}-\eta\
abla_{x}f\left(x_{k}\right)\), \(\eta\) being the step size, which is also referred to as _learning rate_. Gradient descent methods are based on so-called _batches_ employed to calculate the gradient in a single iteration. Commonly, the batch is the entire data set. In addition, a batch can be enormous. A very large batch may cause even a single iteration to take a very long time in computation. By choosing examples from a data set at random, it could estimate (albeit noisily) a big average from a much smaller one. _Stochastic gradient descent_ (SGD) takes this idea to the extreme--it uses solely one example (a batch size of 1) per iteration. Every one computation for all data points, it is called one _epoch_. The term \"stochastic\" implies that the one example comprising each batch is chosen arbitrarily. The mini-batch stochastic gradient descent method (mini-batch SGD) is a compromise between the full-batch iteration and SGD. A mini-batch is usually between 10 and 1000 examples, chosen at random. Mini-batch SGD reduces the amount of noise in SGD but is still more efficient than full-batch. It is well known that the steepest gradient descent faces difficulties in areas where surface curves exhibit different gradients in different dimensions, which frequently occurs around local optima. To reduce the risk of getting stuck in local optima, we take advantage of the _Momentum Method_, which shows analogies to the equations governing the movement of particles in a viscous medium:  
\[ \begin{array}{ll}x_{k+1}&=x_{k}-v_{k}\\ v_{k}&=\gamma v_{k-1}+\eta\
abla_{x}f\left(x_{k}\right)\end{array}  \tag{12}\]  
where the parameter \(\gamma\approx 0.9\) governs the updating of the iterations within the stochastic gradient descent method (SGDM). This approach is commonly employed with the back propagation algorithm,which will be explained later. However, the momentum method is based on the situation where a ball rolling downhill blindly follows the slope. A smarter ball would slow down before the slope goes up again, which is the essence of the _Nesterov Accelerated Gradient_ (NAG) approach [34]. Therefore, the changing of momentum is computed, which is simply the sum of the momentum vector and the gradient vector at the current step. The changing of the Nesterov momentum is then the sum of the momentum vector and the gradient vector at the approximation of the next step:  
\[ \begin{array}{ll}\mathbf{x}_{k+1}&=\mathbf{x}_{k}-\mathbf{v}_{k}\\ \mathbf{v}_{k}&=\gamma\mathbf{v}_{k-1}+\eta\
abla_{\mathbf{x}}f\left(\mathbf{x}_{k}-\gamma\bm {v}_{k-1}\right)\end{array}  \tag{13}\]  
Gradients of complex functions as used in DNN tend to either vanish or explode. These vanishing/exploding problems become more pronounced with increasing complexity of the function. These issues can be alleviated by an adapted learning rate method, named RMSProp, which was suggested by [PERSON] in Lecture 6e of his Coursera Class. The idea is based on a moving average of the squared gradients, which normalizes the gradient. This approach proves effective to balance the step size by decreasing the step for large gradient to avoid explosion while increasing the step for small gradients, which eventually alleviates the vanishment problem:  
\[ \begin{array}{ll}r_{k}&=\left(1-\beta\right)\cdot\left[\
abla_{\mathbf{x}}f\left( \mathbf{x}_{k}\right)\right]^{2}+\beta r_{k-1}\\ \mathbf{v}_{k+1}&=\frac{\eta}{\sqrt{r_{k}}}\
abla_{\mathbf{x}}f\left(\mathbf{x}_{k}\right) \\ \mathbf{x}_{k+1}&=\mathbf{x}_{k}-\mathbf{v}_{k+1}\end{array}  \tag{14}\]  
where the so-called _squared gradient decay factor_\(\beta\) ranges from 0 to 1; we employ the suggested value of \(\beta\approx 0.9\).