## 7 Conclusions  
This paper presents an analysis of the causes of the gradient problem from the perspectives of gradient variance and covariance. It also proposes a new gradient updating method and an evaluation metric to quantify the gradient problem based on statistical theory. The efficacy of the WGN method in alleviating the gradient problem in deep neural networks, particularly recurrent neural networks and networks with long time-dependent characteristics, has been demonstrated through experimentation on multiple public datasets. In addition to suppressing the gradient problem, the WGN method has been shown to accelerate model convergence and improve accuracy. Furthermore, it is our contention that the WGN algorithm requires further examination in the following instances:  
(1) Our findings indicate that the implementation of the WGN method in a single-layer RNN neural network does not result in a notable enhancement in the accuracy rate. However, the LOE index of the model exhibits a considerable improvement. This is due to the monotonous structure of the single-layer Simple-RNN, which produces a permutation effect on the LOE metrics and accuracy.  
(2) In some cases, the use of the gradient normalization method has been observed to occasionally result in a decrease in accuracy or an increase in LOE metrics when applied to smaller datasets. This is primarily attributable to the fact that the batch size is constrained when the dataset is limited in size. This results in an incomplete representation of the overall sample probability within the sample space corresponding to each batch, which in turn gives rise to an increase in the discrepancy between the batches and affects the robustness of the training process. In small-scale datasets where the variability of samples is stronger, the use of gradient normalization may result in an over-adjustment of the gradient, which in turn affects the overall performance of the model. In contrast, in large-scale corpus datasets, the WGN method is more likely to capture the overall characteristics of the gradients and effectively balance the gradient differences between different samples.  
(3) The necessity for the additional computation of the mean and variance of the gradient for each batch inherent to the gradient normalization method also results in an increase in time complexity. It is therefore not recommended that the gradient normalization method be employed in scenarios where there is a high requirement for real-time processing and a low requirement for accuracy.  
In general, the models incorporating WGN appear to yield the most favorable outcomes across a range of tasks. This suggests that the WGN approach has broad applicability and can provide effective support for the training of diverse types of models. In future research, we will continue to investigate the efficacy of WGN and apply it to a broader range of models, including GRU and Transformer.  
Conceptualisation, X.W.; methodology, B.X. and C.L.; software, W.H.; validation, H.L. and X.H.; formal analysis, X.W.; investigation, H.L.; resources, B.X.; data curation, W.H.; writing--original draft preparation, X.H.; writing--review and editing, X.W.; visualization, X.H.; supervision, C.L.; project administration, B.X.; funding acquisition, C.L. All authors have read and agreed to the published version of the manuscript.  
This work was supported by National Natural Science Foundation of China (Grant No. 62106083), the Natural Science Foundation of Xiamen Municipality (Grant No. 3502Z202373035), and the Youth Program of the Natural Science Foundation of Fujian Province of China (Grant No. 2022J05162).  
Not applicable.