# The Method of Restoring Lost Information from Sensors Based on Auto-Associative Neural Networks
## 3 Materials and Methods
Applying the chain rule to calculate the loss function gradients over the ANN parameters, the network weights and bias are updated using gradient descent:  
Figure 1:2  
Figure 1:3  
Steps 2-4 are repeated for each training epoch or until a stopping criterion is reached (e.g., decreasing the loss function to a certain threshold). Using the proposed ANN training algorithm, a hypothesis is proposed for updating the weights in the ANN using gradient descent with \(L_{1}\) and \(L_{2}\) regularization:  
**Hypothesis 1**. : _When using gradient descent with \(L_{1}\) and \(L_{2}\) regularization to update the auto-associative neural network (autoencoder) weights, the network weights will be updated as follows:_  
Figure 1:4  
**Proof of Hypothesis 1. ** Let \(W_{ij}^{(l)}\) be the weight between neurons \(i\) and \(j\) on layer \(l\), \(\alpha\) be the training rate, and \(\lambda_{1}\) with \(\lambda_{2}\) be the regularization coefficients of \(L_{1}\) and \(L_{2}\), respectively. According to the gradient descent algorithm with \(L_{1}\) and \(L_{2}\) regularization, the weights are updated according to the following expression:  
Figure 1:5  
Next, the partial derivatives of the loss function for the weights for updating are derived:  
Figure 1:6  
After loss and activation functions derivatives substitution, the following expression is obtained:  
Figure 1:7  
Thus, according to (11), the ANN weights were updated taking into account the loss function gradient and the \(L_{1}\) and \(L_{2}\) regularization. The derivation of this proof of the hypothesis shows updated ANN weights when using gradient descent with \(L_{1}\) and \(L_{2}\) regularization. In particular, it can be seen that updating the weights includes three components:  
1. The loss function gradient over the weights, which is responsible for correcting the weights taking into account the error between the network output and the input data;
2. \(L_{1}\) regularization, which penalizes the absolute value of the weights, helps in reducing the model complexity, and prevents overfitting;
3. \(L_{2}\) regularization, which penalizes the square of the weight and, also, helps the control model reduce overfitting and complexity. Thus, updating the weights takes into account both the error in the data training and the two types of regularization, which helps in creating a more generalized and robust model. \(\square\)  
Regularization coefficients \(\lambda_{1}\) and \(\lambda_{2}\) must be set manually or selected using cross-validation or optimization methods:  
1. Manually--\(\lambda_{1}\) and \(\lambda_{2}\) are selected by the expert or researcher based on their experience and intuition regarding the problem and data. If a more compressed model is required, then \(\lambda_{1}\) and \(\lambda_{2}\) can be chosen more strongly; if the model is prone to overfitting, then fewer may be selected. 2. Cross-validation is dividing the training data into several parts (folds), training the model on one part, and evaluating its performance on the remaining parts. The process is then repeated for different values of \(\lambda_{1}\) and \(\lambda_{2}\). The values that show the best performance on the test data are selected as optimal. 3. Optimization methods can used to automatically select the \(\lambda_{1}\) and \(\lambda_{2}\) values to minimize some target functionality, for example, the error in the validation set. This process may include optimization techniques such as grid search or surrogate model optimization algorithms. Thus, when choosing values for \(\lambda_{1}\) and \(\lambda_{2}\), it is important to consider the balance between preventing overfitting and maintaining the information content of a model. Also, regularization values may depend on the data nature and size as well as the model complexity.