## 3 Numerical Experiments and Discussion
### Recurrent Neural Network  
Let us consider the model recurrent neural network (RNN) used for the analysis of phrase tone. For details of its architecture and realization, see [[https://python-scripts.com/recurrent-neural-network](https://python-scripts.com/recurrent-neural-network)]([https://python-scripts.com/recurrent-neural-network](https://python-scripts.com/recurrent-neural-network)) (accessed on 16 March 2024). This RNN was realized using the following recurrent relations:  
\[h_{s}=\tanh(W_{xh}x_{s}+W_{hh}h_{s-1}+b_{h}),\quad s=\overline{1,M},\quad y=W_{ hy}h_{M}+b_{y},\]  
where \(M\) is the number of words of vocabulary in the phrase; \(x_{s}\) is a vector, which represents the \(s\)-th word in the phrase; \(h_{s}\) is a vector used for iterations in the hidden layer; \(y\) is the output vector; \(W_{xh},W_{hh}\), \(W_{hy}\) are the matrices of weights; and \(b_{h}\) and \(b_{y}\) are the vectors of biases. The vector of probabilities of the 'good' or 'bad' tone of the phrase was computed as \(\text{softmax}(y)\). The training dataset consisted of 67 phrases from the vocabulary, with 19 unique words. The following dimensions of vectors were used: \(\dim(x)=19\), \(\dim(y)=2\), the dimension of \(h\) was chosen as 64 (the maximum number of words from vocabulary in the phrase; this number can be varied).  
As a result of forward propagation, we obtained a 2D vector of probabilities for the phrase tone, computed with the use of the softmax function. The loss function used for the training of this RNN was computed as  
\[L(X,\theta)=H_{\mu}(\mu,p(X;\theta)),\]  
where \(X\) is a matrix of vectors \(x_{1},\dots,x_{M}\), which represents the phrase with \(M\) words, \(\mu\in\{0,1\}\) is a label of phrase; represented by \(X\); \(p(X)=\text{softmax}(y(X))\) is the probability of the phrase tone; \(H_{\mu}\) is a proper component of a cross-entropy function  
\[H(\
u,p)=-(\
u\log(p)+(1-\
u)\log(1-p));\]  
and \(\theta\in\mathbb{R}^{d}\) is a vector of parameters of RNN. The objective function is written as  
\[f(\theta)=\frac{1}{N}\sum_{i=1}^{N}L(X_{i},\theta),\]  
where \(N=67\) is the size of the training dataset (number of phrases). With all considered dimensions, we minimized the function of \(d=5506\) variables.  
For minimization, we applied deterministic methods, as was considered in the theoretical part of the presented paper and despite the use of stochastic methods in most works on the training of neural networks. The computations were performed with \(h=0.05\), \(\beta_{1}=0.9\)  
Figure 11: Plots of the dependence of error on the iteration number for the logistic regression problem in log–log axes for datasets SONAR (**a**) and CINA0 (**b**). Blue line corresponds to the HBM, red line—to method (4).  
and we tried to vary the value of \(\beta_{2}\) in order to analyze its effect on the convergence. We realized a numerical experiment for 250 random initializations of weights and biases and performed computations for \(3\times 10^{3}\) epochs. In Figure 12, the plots of the dependence of the objective function value on the epoch number averaged at all random initializations are presented for the standard GD (2), HBM (3), and method (4) in the case of \(\beta_{2}=1\). As can be seen, methods with momentum led to a faster convergence in comparison with the standard GD, as mentioned by many authors (e.g., see [19]), and the presence of \(\beta_{2}\) led to a faster convergence to the minimum in practice. In Figure 13, the plots obtained for different values of \(\beta_{2}\) are presented. As can be seen, the value of \(\beta_{2}\) had an effect on the convergence of method (4).  
Figure 12: Plots of the dependence of the objective function value on the epoch number for the problem of RNN training.  
Figure 13: Plots of the dependence of the objective function value on epoch number for the problem of RNN training for method (4) at different values of \(\beta_{2}\).