## 3 Artificial Neural Networks
### Artificial Neural Network Training  
Training a neural network involves optimizing the weights and biases associated with its neurons, specifically for a subset of the dataset known as the training set. The goal is to progressively minimize prediction errors. To achieve this, an error or loss function, along with an algorithm, is introduced. The backpropagation algorithm, which is based on gradient descent methods, is commonly used to reduce the error between the network's output and the desired target.  
In essence, after a forward pass, backpropagation adjusts parameters through a backward pass, fine-tuning the model [49]. It starts with random weight and bias initialization, comparing results to target outcomes, leading to the loss error function. The gradient descent method then calculates partial derivatives to identify influential nodes in minimizing loss, as shown in Equation (2).  
\[\theta_{i}^{(t+1)}=\theta_{i}^{(t)}-\eta\frac{\partial J(\theta_{0},\theta_{1},\theta_{2},...,\theta_{p})}{\partial\theta_{i}} \tag{2}\]  
where  
* \(\theta_{i}^{(t+1)}\) represents the updated weight (or bias) at time \(t+1\).
* \(\theta_{i}^{(t)}\) represents the current weight (or bias) at time \(t\).  
Figure 3: Neural network neuron scheme.  
* \(\eta\) is the learning rate, which controls the step size for weight updates.  
Those nodes receive smaller weights to reduce the overall loss of the model. In other words, it measures how sensitive the loss is to changes in each weight. This is carried out using calculus and gradients to determine the direction and magnitude of the steepest increase in the loss. The learning rate controls the update magnitude, with smaller rates leading to more stable convergence, but more iterations may be required. Larger rates can speed up convergence but risk overshooting the minimum. In summary, during each training cycle (epoch) of the neural network, the backpropagation algorithm adjusts weights by subtracting derivatives scaled by a \"learning rate\" (which avoids abrupt variations of the weights). This iterative optimization process continues until the error function reaches a desirable threshold for the application.  
To perform the explained training process of the neural network, the authors decided to use the Levenberg-Marquardt algorithm. From the foundational principles of standard backpropagation with gradient descent, the Levenberg-Marquardt algorithm emerged as an innovative optimization method for neural network training. This algorithm introduces a novel approach by incorporating the Hessian matrix, capturing the second-order partial derivatives of the cost function with respect to network weights. It dynamically adjusts a damping parameter to modulate the step size in weight updates, adapting to the local curvature of the cost function's surface. The resulting weight updates become a function of the gradient, Jacobian matrix, error vector, and Hessian matrix, enabling faster convergence and improved stability during training. Its functioning is summarized in Equation (3).  
\[\theta^{(t+1)}=\theta^{(t)}-\left[J^{T}\cdot J+\lambda\cdot\text{diag}(J^{T} \cdot J)\right]^{-1}\cdot J^{T}\cdot e \tag{3}\]  
where  
* \(\theta^{(t+1)}_{i}\) represents the updated weight (or bias) at time \(t+1\).
* \(\theta^{(t)}_{i}\) represents the current weight (or bias) at time \(t\).
* \(J\) is the Jacobian matrix, representing the first-order partial derivatives of the cost function with respect to the weights.
* \(e\) is the error vector, representing the difference between the network's predictions and the target values.
* \(diag(J^{T}*J)\) represents the diagonal elements of the Hessian matrix.  
This method is particularly valuable when training neural networks with relatively few weights and is recognized for its ability to expedite the optimization process in various scientific and engineering domains.