## 3 Improved Stochastic-Batch Gradient Pile Descent
### The Improvement Method Based on the Gradient Pile (GP)  
In the traditional gradient descent (GD) algorithm, the optimization variable is updated using the negative gradient as the correction term in each iteration. For an unconstrained optimization model, the \(k\)-th iteration of the traditional GD algorithm is performed using the following formula:  
\[\text{min}\mathbf{H}(x) \tag{28}\]  
\[x_{k+1}=x_{k}+\lambda_{1}(-\
abla_{x_{k}}\mathbf{H}) \tag{29}\]  
In Equations (28) and (29), \(\mathbf{H}\) is the objective function of the optimization model. \(x\) is the optimization variable. \(x_{k}\) represents the value of x at the \(k\)-th iteration. \(-\
abla_{x_{k}}\mathbf{H}\) is the negative gradient of \(\mathbf{H}\) with respect to \(x\). \(\lambda_{1}\) is the step size used in the iteration. The traditional GD algorithm guides the optimization variable to find points where the gradient is zero and uses a gradient equal to zero as the convergence criterion. However, during parameter training, there are many local optimal points and saddle points in the solution space where the gradient is zero. If the gradient equal to zero is the only convergence criterion, the GD algorithm may get stuck in local optimal points and saddle point regions during the training process, reducing training accuracy. To address this problem, this paper proposes an improvement based on the accumulation of gradients as the correction term. The \(k\)-th iteration step is given by  
\[\mathbf{a}_{k}=-\
abla_{x_{k}}\mathbf{H} \tag{30}\]  
\[\mathbf{v}_{k}=\lambda_{2}\mathbf{v}_{k-1}+\lambda_{3}\mathbf{a}_{k} \tag{31}\]  
Figure 2: A conceptual illustration of the improved stochastic–batch gradient descent (S-BGD) search method. \[x_{k+1}=x_{k}+\lambda_{4}\mathbf{v}_{k} \tag{32}\]  
In Equations (30)-(32), \(\mathbf{a}\) and \(\mathbf{v}\), respectively, represent the negative gradient of \(\mathbf{x}\) and its accumulation. \(\lambda_{2}\) and \(\lambda_{3}\) are weight parameters, and \(\lambda_{4}\) is the iteration step size. To ensure the convergence of the iterative computation, the accumulation of gradients should exhibit a decaying trend during the iteration process. Therefore, it is specified that \(\lambda_{2}<1\). It can be observed that using the gradient accumulation as the correction term and considering the gradient accumulation as 0 as the convergence criterion not only maintains the descent rate of the negative gradient but also effectively avoids getting trapped in saddle points and local optima. By doing so, it enables the search for the global optimal solution and thus enhances the optimization capability of the GD algorithm. The search process based on gradient accumulation is illustrated in Figure 3. In the figure, a ball starts rolling down from the top of the hill, where the gradient accumulation is represented as \(\mathbf{v}\) and the negative gradient as \(\mathbf{a}\). The negative gradient acts as an acceleration, guiding the motion of the ball. At the extremum point A and the saddle point B, the values of \(\mathbf{a}\) become 0. According to the traditional GD algorithm, the computation would end at either point A or B. However, as seen in Figure 3, the optimal point is actually C, not A or B. Thus, the traditional GD algorithm gets trapped in a local optimum or a saddle point. On the other hand, using the gradient accumulation-based search method proposed in this paper, when the ball reaches point A or B, it continues to move forward to point C, effectively avoiding the problem of getting stuck in local points or saddle points during the search process. In conclusion, this paper proposes a comprehensive improvement in parameter training strategy by employing both the S-BGD and GP methods. This approach enhances training efficiency while effectively avoiding convergence to local points and saddle points. We refer to this enhanced method as the stochastic-batch gradient pile descent (S-BGPD) method.