SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aleksandr Beznosikov
Innopolis University, Skoltech, MIPT, Yandex
Sergey Samsonov
Marina Sheshukova
MIPT, Skoltech, IITP RAS
Alexey Naumov
Eric Moulines
Ecole polytechnique
This paper delves into stochastic optimization problems that involve Markovian noise. We present a unified approach for the theoretical analysis of first-order gradient methods for stochastic optimization and variational inequalities. Our approach covers scenarios for both non-convex and strongly convex minimization problems. To achieve an optimal (linear) dependence on the mixing time of the underlying noise sequence, we use the randomized batching scheme, which is based on the multilevel Monte Carlo method. Moreover, our technique allows us to eliminate the limiting assumptions of previous research on Markov noise, such as the need for a bounded domain and uniformly bounded stochastic gradients. Our extension to variational inequalities under Markovian noise is original. Additionally, we provide lower bounds that match the oracle complexity of our method in the case of strongly convex optimization problems.
Stochastic gradient methods are an essential ingredient for solving various optimization problems, with a wide range of applications in various fields such as machine learning [36, 37], empirical risk minimization problems [96], and reinforcement learning [93, 85, 69]. Various stochastic gradient descent methods (SGD) and their accelerated versions [75, 31] have been extensively studied under different statistical frameworks [17, 97]. The standard assumption for stochastic optimization algorithms is to consider independent and identically distributed noise variables. However, the growing usage of stochastic optimization methods in reinforcement learning [10, 87, 25] and distributed optimization [63, 18, 65] has led to increased interest in problems with Markovian noise. Despite this, existing theoretical works that consider Markov noise have significant limitations, and their analysis often results in suboptimal finite-time error bounds.
Our research aims to fill the gap in the existing literature on the first-order Markovian setting. By focusing on uniformly geometrically ergodic Markov chains, we obtain finite-time complexity bounds for achieving  $\varepsilon$ -accurate solutions that scale linearly with the mixing time of the underlying Markov chain. Our approach is based on careful applications of randomized batch size schemes and provides a unified view on both non-convex and strongly convex minimization problems, as well as variational inequalities.
Our contributions. Our main contributions are the following:
$\diamond$  Accelerated SGD. We provide the first analysis of SGD, including the Nesterov accelerated SGD method, with Markov noise without the assumption of bounded domain and uniformly bounded stochastic gradient estimates. Our results are summarised in Table I and Section 2.1 and cover both strongly convex and non-convex scenarios. Our findings for non-convex minimization problems complement the results obtained in [21].
Lower bounds. In Section 2.2 we give the lower bounds showing that the presence of mixing time in the upper complexity bounds is not an artefact of the proof. This is consistent with the results reported in [71].
$\diamond$  Extensions. In Section 2.4 we provide, as far as we know, the first analysis for variational inequalities with general stochastic Markov oracle, arbitrary optimization set, and arbitrary composite term. Our finite-time performance analysis provides complexity bounds in terms of oracle calls that scale linearly with the mixing time of the underlying chain, which is an improvement over the bounds obtained in [99] for the Markov setting.
Related works. Next, we briefly summarize the related works.
$\diamond$  Stochastic gradient methods. Numerous research papers have reported significant improvements achieved by accelerated methods for stochastic optimization with stochastic gradient oracles involving independent and identically distributed (i.i.d.) noise. These methods have been extensively studied in theory [44, 14, 16, 58, 61, 26, 30, 97, 94, 3, 39, 102] and have shown practical success [55, 91]. The finite-time analysis of first-order methods in i.i.d. noise settings has been extensively studied by many authors, as discussed in [59] and references therein. In Table I we include only some important results because i.i.d. setting is not in the interest of this paper.
While the literature on i.i.d. noise is extensive, existing research on the first-order Markovian setting is relatively sparse. In this study, we focus on Markov chains that are uniformly geometrically ergodic, and we refer the reader to Section 2 for detailed definitions. We note that the complexity bounds which scale linearly with the mixing time of the underlying general Markov chain are currently available only for general convex and non-convex minimization problems. Namely, [23] has investigated a version of the ergodic mirror descent algorithm that yields optimal convergence rates for Lipschitz, general convex and non-convex problems. Recently, [21] proposed a random batch size algorithm that adapts to the mixing time of the underlying chain for non-convex optimization with a compact domain. In particular, [21, Theorem 4.3] yields optimal complexity rates in terms of the number of oracle calls required for non-convex problems, which is consistent with the results obtained in [23]. Unlike previous studies, this method is insensitive to the mixing time of the noise sequence.
For the general case of Markovian noise the finite-time analysis of non-accelerated SGD-type algorithms was carried out in [90] and [19]. However, [90] heavily relies on the bounded domain assumption and uniformly bounded stochastic gradient oracles, while its bound in [90, Theorem 5] has a suboptimal dependence on the mixing time of the underlying chain, see Table. Additionally, [90] does not cover the strongly convex setting. On the other hand, [19] covers both non-convex and strongly convex settings, but the bounds of [19, Theorem 1] has terms that are exponential in the mixing time, and a careful examination reveals suboptimal dependence on the initial condition for strongly convex problems when SGD is applied.
In the study of Nesterov-accelerated SGD with Markovian noise, the authors of [20] considered the use of a batch size of 1 and achieved a rate of forgetting the initial condition that matches that of the i.i.d. noise setting. However, their result is suboptimal in terms of the variance terms in both non-convex and strongly convex settings, as detailed in Table I. We emphasize that the case of unbounded gradient oracles with Markov noise is not treated in contrast to the i.i.d. setup [97, 62].
The above papers deal with general Markovian noise optimization. But there are also results that deal with Markovian stochasticity with a finite state space. Here we can highlight the work [28], where the author gives quite extensive results and achieves linear scaling by mixing time in the non-convex as well as strongly convex cases. Recently, numerous papers have appeared dealing with the special scenario of distributed optimization [89]. [99] investigates the generalization and stability of Markov SGD with special attention to the excess variance guarantees. We note that first, these algorithms only need to deal with a very special case of Markov gradients, and second, the corresponding dependence on the mixing time of the Markov chain is again quadratic. At the same time, there exist particular results, e.g. [71], which provide a lower bound for the particular finite sum problems in the Markovian setting.
$\diamond$  Variational inequalities. Variational inequalities [29] have been an active area of research in applied mathematics for more than half a century [78, 41, 86]. VI cover important special cases, e.g., minimization over a convex domain, saddle point or min-max and fixed point problems. Computational game theory [29], robust [7] and nonsmooth [73, 72] optimization, supervised [51, 4] and unsupervised [103, 5] learning, image denoising [27, 11]. In the last 5 years, variational inequalities and their special cases have attracted much interest in the machine learning community
due to new connections to reinforcement learning [79, 50], adversarial training [64], and GANs [15, 33, 66, 12, 60, 82].
Variational inequalities (VI) and saddle point problems have their own well-established theory and methods. Unlike minimization problems, solving variational inequalities doesn't rely on (accelerated) gradient descent. Instead, the extragradient method [57], various modified versions [72, 42], or similar techniques [95] are recommended as the basic and theoretically optimal methods. While deterministic methods have long been used for solving variational inequalities, stochastic methods have gained importance only in the last 15 years, following pioneering works by [49, 52]. We summarise the results on methods for stochastic variational inequalities with the Lipschitz operator and smooth stochastic saddle point problems in Table 2. The number of papers dealing with stochastic VIs and saddle point problems is small compared to those dealing with stochastic optimization, we include in Table 2 papers with the i.i.d. noise (which we do not do for stochastic optimization). The only competing work dealing with Markovian noise in saddle point problems considers the finite sum problem and thus the finite Markov chain [99], therefore we do not include it in Table 2. Moreover, the results from [99] has much worse oracle complexity guarantees  $\mathcal{O}(\tau^2/\varepsilon^2)$  in terms of  $\tau$ . There are more papers dealing with stochastic finite-sum variational inequalities or saddle point problems, but in the i.i.d. setting [12, 80, 104, 2, 8]. We also do not consider in Table 2 because of the difference in the stochastic oracle structure. It is important to note that, unlike most previous works, we consider the most general formulation of VI itself for an arbitrary optimization set and composite term.
Notations and definitions. Let  $(Z, \mathrm{d}_Z)$  be a complete separable metric space endowed with its Borel  $\sigma$ -field  $\mathcal{Z}$ . Let  $(Z^{\mathbb{N}}, Z^{\otimes \mathbb{N}})$  be the corresponding canonical process. Consider the Markov kernel  $Q$  defined on  $Z \times \mathcal{Z}$ , and denote by  $\mathbb{P}_{\xi}$  and  $\mathbb{E}_{\xi}$  the corresponding probability distribution and the expected value with initial distribution  $\xi$ . Without loss of generality, we assume that  $(Z_k)_{k \in \mathbb{N}}$  is the corresponding canonical process. By construction, for any  $A \in \mathcal{Z}$ , it holds that  $\mathbb{P}_{\xi}(Z_k \in A | Z_{k-1}) = Q(Z_{k-1}, A)$ ,  $\mathbb{P}_{\xi}$ -a.s. If  $\xi = \delta_z$ ,  $z \in Z$ , we write  $\mathbb{P}_z$  and  $\mathbb{E}_z$  instead of  $\mathbb{P}_{\delta_z}$  and  $\mathbb{E}_{\delta_z}$ , respectively. For  $x^1, \ldots, x^k$  being the iterates of any stochastic first-order method, we denote  $\mathcal{F}_k = \sigma(x^j, j \leq k)$  and write  $\mathbb{E}_k$  as an alias for  $\mathbb{E}[\cdot | \mathcal{F}_k]$ . We also write  $\mathbb{N}^* := \mathbb{N} \setminus \{0\}$ . For the sequences  $(a_n)_{n \in \mathbb{N}}$  and  $(b_n)_{n \in \mathbb{N}}$  we write  $a_n \lesssim b_n$  if there exists a constant  $c$  such that  $a_n \leq c b_n$  for all  $n \in \mathbb{N}$ .
Table 1: This table summarizes our results on first-order method with Markovian noise. The columns of the table indicate whether the authors consider optimization over bounded domain, potentially unbounded gradients, and whether or not they assume additional restrictions on the Markovian noise (finite state space or reversibility). For ease of comparison we provide the respective results on SGD and ASGD (accelerated SGD) in the i.i.d. setting.
Assumptions. In this paper we study the minimization problem
$$
\min  _ {x \in \mathbb {R} ^ {d}} f (x) := \mathbb {E} _ {Z \sim \pi} [ F (x, Z) ], \tag {1}
$$
where the access to the function  $f$  and its gradient is available only through the (unbiased) noisy oracle  $F(x,Z)$  and  $\nabla F(x,Z)$ , respectively. In the following presentation we impose at least one of the following regularity constraint on the underlying function  $f$  itself:
Table 2: This table summarizes the findings on methods for solving stochastic (strongly) monotone variational inequalities with a Lipschitz operator and (un)bounded stochasticity. The columns of the table indicate whether the authors consider variational inequalities or only certain saddle point problems, the arbitrariness of the sets, and the use of additional composite terms. The columns on stochasticity provide information on the assumptions made with respect to the stochastic operator, such as bounded variance and the Markovian noise setting. Note that with the exception of our work, all other studies assume the independent noise.
Statement	Stochasticity
VI?	Any set?	Composite?	Unbounded?	Markovian?	Oracle complexity
Strongly monotone	SPEG 33.42	✓	✓	X	X	X	O(L/μ log ||x0-x*||2+σ2/μ2ε)(1)
SEG 53	✓	✓	X	X	X	O(||x0-x*||2/ε + B2+σ2+(B+σ)(1+LD)/σ2ε)
SS-SEG 68.38	✓	✓	✓	✓	X	O(L/μ log ||x0-x*||2/ε + σ2/μ2ε)
SEG 9	X	✓	X	X	X	O(L/μ log ||x0-x*||2/ε + σ2/μ2ε)
DSEG 43	✓	X	X	✓	X	O[(L2/μ2ε)]3(2)
UEG 38	✓	X	X	✓	X	O((L+Δ/μ + Δ2/μ2) log ||x0-x*||2/ε + σ2/μ2ε)
SGDA 8	✓	✓	✓	X	X	O(L2/μ2 log ||x0-x*||2/ε + σ2/μ2ε) (3)
REG (ours)	✓	✓	✓	✓	✓	O(τ·[L+Δ/μ + Δ2/μ2) log ||x0-x*||2/ε + σ2/μ2ε])
Monotone	SMP 52	✓	✓	X	X	X	O(LD2/ε + σ2Δ2/ε)
VR-SEG 45	✓	✓	X	✓	X	O((σ+Δ)8D4/ε2 + D4/ε2)
IPM 46	✓	✓	✓	✓	X	O(0(L4D4/ε2 + σ2D4/ε2))
SS-SEG 68	✓	✓	✓	✓	X	O(L2D4/ε + σ4/L2ε2)
SEG 9	X	✓	X	X	X	O(LD2/ε + σ2Δ2/ε2)
REG (ours)	✓	✓	✓	✓	✓	O(τ·[LD2/ε + σ2D2/ε2 + Δ2D4/ε2])
notation:  $\mu =$  constant of strong monotonicity of operator  $F$ ,  $L =$  Lipschitz constant of  $F$ ,  $B =$  uniform bound of  $F$ ,  $D =$  uniform bound of iterations  $x^k$ ,  $x^0 =$  starting point,  $x^* =$  solution,  $\Delta$  and  $\sigma =$  stochasticity parameters (see A [52,33,42,53,9], take  $\Delta = 0$ ),  $\sigma_* =$  stochasticity parameter in  $x^*$  (see [68]),  $\tau =$  mixing time of the chain (see A [3],  $\varepsilon =$  accuracy of the solution. (1) give results with stepsize as a parameter, we choose it according to Section 3 from [88]. (2) consider A [7] but do not provide explicit rates if  $\Delta \neq 0$  (see also [38], Table 1]). (3) consider the cocoercive case, for which in general  $\ell = L^2/\mu$ .
A 1. The function  $f$  is  $L$ -smooth on  $\mathbb{R}^d$  with  $L > 0$ , i.e., it is differentiable and there is a constant  $L > 0$  such that the following inequality holds for all  $x, y \in \mathbb{R}^d$ :
$$
\| \nabla f (x) - \nabla f (y) \| \leq L \| x - y \|.
$$
A 2. The function  $f$  is  $\mu$ -strongly convex on  $\mathbb{R}^d$ , i.e., it is continuously differentiable and there is a constant  $\mu > 0$  such that the following inequality holds for all  $x, y \in \mathbb{R}^d$ :
$$
(\mu / 2) \| x - y \| ^ {2} \leq f (x) - f (y) - \langle \nabla f (y), x - y \rangle . \tag {2}
$$
Next we specify our assumptions on the sequence of noise variables  $\{Z_i\}_{i=0}^{\infty}$ . We consider here the general setting of  $\{Z_i\}_{i=0}^{\infty}$  being a time-homogeneous Markov chain. Such problems naturally arise in stochastic optimization. In the empirical risk minimization problems it naturally appears in the context of non-random minibatch choice. Indeed, a random choice of a batch number may lose to a non-random one, see [67, 56]. A wide range of problems dealing with Markovian noise is spawned by the reinforcement learning methods. The usual MDP setting falls naturally inside this paradigm, moreover, the analysis of non-tabular RL problems requires to deal with the general state-space Markov noise. Here the potential range of applications include the policy evaluation methods, such as the temporal difference methods [92], and policy optimization algorithms, such as policy gradient family, e.g. the celebrated REINFORCE algorithm [100].
We denote by  $\mathbf{Q}$  the Markov kernel corresponding to the sequence  $\{Z_i\}_{i=0}^{\infty}$  and impose the following assumption on the mixing properties of  $\mathbf{Q}$ :
A 3.  $\{Z_i\}_{i=0}^{\infty}$  is a stationary Markov chain on  $(\mathbb{Z}, \mathcal{Z})$  with Markov kernel  $Q$  and unique invariant distribution  $\pi$ . Moreover,  $Q$  is uniformly geometrically ergodic with mixing time  $\tau \in \mathbb{N}$ , i.e., for every  $k \in \mathbb{N}$ ,
$$
\Delta \left(\mathrm {Q} ^ {k}\right) = \sup  _ {z, z ^ {\prime} \in \mathsf {Z}} (1 / 2) \| \mathrm {Q} ^ {k} \left(z, \cdot\right) - \mathrm {Q} ^ {k} \left(z ^ {\prime}, \cdot\right) \| _ {\mathsf {T V}} \leq (1 / 4) ^ {\lfloor k / \tau \rfloor}. \tag {3}
$$
The assumption A is classical in the literature on optimization methods with Markovian noise and has been considered in particular in recent works [90, 21, 20]. In particular, this assumption covers finite state-space Markov chains with irreducible and aperiodic transition matrix considered in
[28]. Yet our definition of the mixing time  $\tau$  is more classical in the probability literature [81], and is slightly different from the one considered e.g. in [28, 65]. Next we specify our assumptions on stochastic gradient:
A 4. For all  $x \in \mathbb{R}^d$  it holds that  $\mathbb{E}_{\pi}[\nabla F(x,Z)] = \nabla f(x)$ . Moreover, for all  $z \in \mathbb{Z}$  and  $x \in \mathbb{R}^d$  it holds that
$$
\left\| \nabla F (x, z) - \nabla f (x) \right\| ^ {2} \leq \sigma^ {2} + \delta^ {2} \| \nabla f (x) \| ^ {2}. \tag {4}
$$
The assumption A4 resembles the strong growth condition [97], which is classical for the overparametrized learning setup [97, 98]. The main difference is that A4 concerns the almost sure bound in (4), which is unavoidable when dealing with uniformly geometrically ergodic Markovian noise A3. Note that it is possible that the quantity  $\delta^2$  in (4) is not instance-independent and scales with the ratio  $L / \mu$  from A1-A2 in the particular problems. With the assumptions A3 and A4 we can prove the result on the mean squared error of the stochastic gradient estimate computed over batch size  $n$  under arbitrary initial distribution. This result is summarized below in Lemma 1.
Lemma 1. Assume  $A$  and  $A$ . Then, for any  $n \geq 1$  and  $x \in \mathbb{R}^d$ , it holds that
$$
\mathbb {E} _ {\pi} \left[ \| n ^ {- 1} \sum_ {i = 1} ^ {n} \nabla F (x, Z _ {i}) - \nabla f (x) \| ^ {2} \right] \leq \frac {8 \tau}{n} \left(\sigma^ {2} + \delta^ {2} \| \nabla f (x) \| ^ {2}\right). \tag {5}
$$
Moreover, for any initial distribution  $\xi$  on  $(Z,\mathcal{Z})$ , that
$$
\mathbb {E} _ {\xi} \left[ \| n ^ {- 1} \sum_ {i = 1} ^ {n} \nabla F (x, Z _ {i}) - \nabla f (x) \| ^ {2} \right] \leq \frac {C _ {1} \tau}{n} \left(\sigma^ {2} + \delta^ {2} \| \nabla f (x) \| ^ {2}\right), \tag {6}
$$
where  $C_1 = 16(1 + \frac{1}{\ln^24})$
Proof. We first prove (5). Note that due to [81 Proposition 3.4] the Markov kernel  $\mathrm{Q}$  under A admits a positive pseudospectral gap  $\gamma_{ps} > 0$  such that  $1 / \gamma_{ps} \leq 2\tau$ . Thus, applying the statement of [81 Theorem 3.2], we get under A that
$$
\mathbb {E} _ {\pi} \left[ \| n ^ {- 1} \sum_ {i = 1} ^ {n} \nabla F (x, Z _ {i}) - \nabla f (x) \| ^ {2} \right] \leq \frac {4 \mathbb {E} _ {\pi} \left[ \| \nabla F (x , Z _ {1}) - \nabla f (x) \| ^ {2} \right]}{n \gamma_ {p s}} \leq \frac {8 \tau}{n} \left(\sigma^ {2} + \delta^ {2} \| \nabla f (x) \| ^ {2}\right).
$$
To prove the second part we use the maximal exact coupling construction and follow, e.g., [24, Theorem 1]. The complete proof is given in Appendix B.1.
The proof of Lemma 1 simplifies the arguments in [21, Lemma 4] and allows us to obtain tighter values for the constants when dealing with the randomized batch size. Note that it is especially important to have the result for MSE under arbitrary initial distribution  $\xi$ , since in the proofs of our main results we will inevitably deal with the conditional expectations w.r.t. the previous iterate. We provide more details on the bias and variance of the Markov SGD gradients in the next section.
2.1 Accelerated method
We begin with a version of Nesterov accelerated SGD with randomized batch size, described in Algorithm I. Due to the unboundedness of the stochastic gradient variance (see A 4), using of the classical Nesterov accelerated method [76, Section 2.2.] does not give the desired result, it is necessary to introduce an additional momentum [74, 97]. We use our own version, but partially similar to [74, 97]. The main feature of Algorithm I is that the number of samples used during the  $k$ -th gradient computation scales as  $2^{J_k}$ , where  $J_{k}$  comes from a truncated geometric distribution. The truncation parameter needs to be adopted (see Theorem 1) in order to control the computational complexity of the algorithm.
Randomized batch size allows for efficient bias reduction in the stochastic gradient estimates and can be seen as a particular case of the so called multilevel MCMC [35, 34]. In the optimization context this approach was successfully used by [21] for the non-convex problems. Indeed, this bias naturally appears under the Markovian stochastic gradients oracles. It is easy to see that, with the counter  $T^{k}$  defined in Line 9 we have
$$
\mathbb {E} _ {k} \left[ \nabla F \left(x ^ {k}, Z _ {T ^ {k} + i}\right) \right] \neq \nabla f \left(x ^ {k}\right).
$$
Below we show how the bias of the gradient estimate scales with the truncation parameter  $M$ . The statement of Lemma 2 yields that the gradient estimates  $g_{k}$  introduced above have the bias, which decreases quadratically with  $M$ .
Lemma 2. Assume  $A \boxdot$  and  $A \boxdot$ . Then for the gradient estimates  $g^k$  from Algorithm [1] it holds that  $\mathbb{E}_k[g^k] = \mathbb{E}_k[g_{\lfloor \log_2 M \rfloor}^k]$ . Moreover,
$$
\mathbb {E} _ {k} \left[ \| \nabla f (x ^ {k}) - g ^ {k} \| ^ {2} \right] \lesssim \left(\tau B ^ {- 1} \log_ {2} M + \tau^ {2} B ^ {- 2}\right) \left(\sigma^ {2} + \delta^ {2} \| \nabla f (x ^ {k}) \| ^ {2}\right),
$$
$$
\| \nabla f (x ^ {k}) - \mathbb {E} _ {k} [ g ^ {k} ] \| ^ {2} \lesssim \tau^ {2} M ^ {- 2} B ^ {- 2} (\sigma^ {2} + \delta^ {2} \| \nabla f (x ^ {k}) \| ^ {2}).
$$
Algorithm 1 Randomized Accelerated GD
1: Parameters: stepsize  $\gamma > 0$ , momentum  $\theta, \eta, \beta, p$ , number of iterations  $N$ , batchsize limit  $M$
2: Initialization: choose  $x^0 = x_f^0$
3: for  $k = 0,1,2,\ldots ,N - 1$  do
4:  $x_{g}^{k} = \theta x_{f}^{k} + (1 - \theta)x^{k}$
5: Sample  $\tilde{J}_k\sim$  Geom (1/2)
6:  $g^{k} = g_{0}^{k} + \left\{ \begin{array}{ll}2^{J_{k}}\left(g_{J_{k}}^{k} - g_{J_{k} - 1}^{k}\right), & \text{if } 2^{J_{k}} \leq M \\ 0, & \text{otherwise} \end{array} \right.$  with  $g_{j}^{k} = 2^{-j}B^{-1}\sum_{i=1}^{2^{j}B}\nabla f(x_{g}^{k}, Z_{T^{k+i}})$ .
7:  $x_{f}^{k + 1} = x_{g}^{k} - p\gamma g^{k}$
8:  $x^{k + 1} = \eta x_f^{k + 1} + (p - \eta)x_f^k +(1 - p)(1 - \beta)x^k +(1 - p)\beta x_g^k$
9:  $T^{k+1} = T^k + 2^{J_k} B$
The proof and the statement with explicit constants are given in Appendix B.2. Note that the Lemma 2 is a natural counterpart of the deterministic bound Lemma 1. Moreover, it gives the idea of the trade-off between the parameters  $B$  and  $M$ . Namely, the expected number of oracle calls to compute  $g_{k}$  is  $\mathcal{O}(B\log_2(M))$  with the bias scaling as  $M^{-2}$ . Thus the increase of  $M$  drastically reduced the bias with only a logarithmic payment in variance. At the same time, gradient variance scales as  $(\tau /B)^{2}$ , but the increase of  $B$  is much more expensive for the computational cost of the whole procedure. Taking into account the considerations above, we can prove the following result:
Theorem 1. Assume  $A \sqcup -A \sqcup$  Let problem (1) be solved by Algorithm (1). Then for any  $b \in \mathbb{N}^*$ ,  $\gamma \in (0; \frac{3}{4L}]$ , and  $\beta, \theta, \eta, p, M, B$  satisfying
$$
p \simeq (1 + (1 + \gamma L) [ \delta^ {2} \tau b ^ {- 1} + \delta^ {2} \tau^ {2} b ^ {- 2} ]) ^ {- 1}, \quad \beta \simeq \sqrt {p ^ {2} \mu \gamma}, \quad \eta \simeq \sqrt {\frac {1}{\mu \gamma}},
$$