## 2 The Theoretical Basis of the Algorithms Used in This Research
### Dnn  
DNNs consist of multiple hidden layers of interconnected neurons. Training a DNN involves adjusting the weights and biases of the neurons in each layer to minimize a loss function, which measures how well the network's output matches the desired output. This measurement is typically performed using optimization algorithms such as Stochastic Gradient Descent, AdaGrad [26], Adam [27], RMSProp [28], and Adadelta [29]. For example, in a two-layer neural network with input neurons \(\vec{x}=(x_{1},\dots,x_{d})\in\mathbb{R}^{d}\)weights \(\overrightarrow{w}=(w_{1},\ldots,w_{d})\in\mathbb{R}^{d}\), and bias \(\beta_{j}\), the value of the hidden neuron is calculated as follows [30]:  
\[h=\sigma(w_{1}x_{1}+w_{2}x_{2}+\cdots+w_{d}x_{d}+\beta)=\sigma\Big{(}w^{T}x+ \beta\Big{)} \tag{2}\]  
where \(\sigma\) is a non-linear function such as Sigmoid function, Rectified linear unit, Hyperbolic Tangent, Exponential linear unit, and many other non-linear activation functions. The mapping is performed using a two-layer neuron network with \(m\) hidden neurons and \(1\) output neuron. \[\sigma\Big{(}b_{k}^{T}x+c_{k}\Big{)},k=1,2,\ldots,m \tag{3}\]  
\[y=\sum_{k=1}^{m}a_{k}h_{k}+a_{0} \tag{4}\]  
A key factor for DNNs is determining the appropriate architecture. If the network has \(L\) layers, then it contains \(n_{l}\) neurons. In general, the \(n_{L}\) output neurons depend on the \(n_{l}\) input neurons through the function \(F:\mathbb{R}^{n1}\longrightarrow\mathbb{R}^{nL}\), whose analytic expression is unknown, and, therefore, the algorithm will be approximate. However, a fully connected deep neural network with L hidden layers is computed as follows:  
\[h_{k}^{(1)}=\sigma\Big{(}\Big{(}b_{k}^{(1)}\Big{)}^{T}x+\theta_{k}^{(1)}\Big{)},k=1,2,\ldots,m^{(1)}, \tag{5}\]  
\[h_{k}^{(l+1)}=\sigma\Big{(}\Big{(}b_{k}^{(l+1)}\Big{)}^{T}h^{l}+\theta_{k}^{( l+1)}\Big{)},l=1,2,\ldots,L-1,k=1,2,\ldots,m^{(l+1)} \tag{6}\]  
\[y=\sum_{k=1}^{m^{(L)}}a_{k}h_{k}^{(l)}+a_{0} \tag{7}\]  
The loss function is a crucial component in the training phase of the neural network. Training in neural networks corresponds to minimizing the loss function during training, which is a type of optimization where the goal is to find the optimal set of weights and biases. Therefore, minimizing the loss function means bringing the predicted output closer to the actual output. During each training iteration, the neural network calculates the gradients of the loss function with respect to the weights and biases using Back-Propagation [31]. Let \(f(x;\theta)\) be a fully connected neural network with \(L\) hidden layers:  
\[f(x;\theta)=W_{L}\sigma(W_{L1}\sigma(\ldots W_{1}\sigma(W_{0}x)\ldots)) \tag{8}\]  
Then,  
\[\theta=\{W_{0},W_{1},\ldots,W_{L}\} \tag{9}\]  
Now, for data sample \((x,y)\) and a loss function \(l(f(x;\theta),y)\), let us calculate the \(\frac{\partial E(f(x;\theta),y)}{\partial w_{l}}\) for some \(l\) by the chain rule (derivative of the error with respect to each weight) [32]. In Equation (10), \(E\) is the symbol of the error function. \[\frac{\partial E}{\partial w_{l}}=\frac{\partial E}{\partial f}.\frac{ \partial f}{\partial h_{l+1}}.\frac{\partial h_{l+1}}{\partial w_{l}} \tag{10}\]  
Since \(\frac{\partial E}{\partial w_{l}}\) must be calculated for any \(l\), it is necessary to obtain \(\frac{\partial f}{\partial h_{l}}\). Therefore, we have:  
\[\frac{\partial f}{\partial h_{l}}=\frac{\partial f}{\partial h_{L}}.\left( \prod_{j=L-1}^{l}\frac{\partial h_{j+1}}{\partial h_{j}}\right)=\frac{ \partial f}{\partial h_{l+1}}.\frac{\partial h_{l+1}}{\partial h_{l}} \tag{11}\]Hence, instead of evaluating \(\prod_{j=L-1}^{l}\frac{\partial h_{j+1}}{\partial h_{j}}\) separately for all \(l\), it can be inversely calculated from \(l=L\) to \(l=1\). Moreover, in each layer, we simply multiply to get \(\frac{\partial f}{\partial h_{l}}\).