SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuki Takezawa, Han Bao, Kenta Niwa, Ryoma Sato, and Makoto Yamada. Momentum tracking: Momentum acceleration for decentralized deep learning on heterogeneous data. arXiv preprint arXiv:2209.15505, 2022.
Hanlin Tang, Xiangru Lian, Chen Yu, Tong Zhang, and Ji Liu. DoubleSqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression. In International Conference on Machine Learning, 2020.
Alexander Tyurin and Peter Richtárik. Dasha: Distributed nonconvex optimization with communication compression, optimal oracle complexity, and no client synchronization. arXiv preprint arXiv:2202.01268, 2022.
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 2019.
Jun-Kun Wang and Jacob Abernethy. Quickly finding a benign region via heavy ball momentum in non-convex optimization. arXiv preprint arXiv:2010.01449, 2021.
Yujia Wang, Lu Lin, and Jinghui Chen. Communication-compressed adaptive gradient method for distributed nonconvex optimization. In International Conference on Artificial Intelligence and Statistics, pages 6292-6320. PMLR, 2022.
Tiannan Xiao and Guoguo Yang. A convergence study of sgd-type methods for stochastic optimization. arXiv preprint arXiv:2211.06197, 2022.
Cong Xie, Shuai Zheng, Oluwasanmi Koyejo, Indranil Gupta, Mu Li, and Haibin Lin. CSER: Communication-efficient SGD with error reset. In Advances in Neural Information Processing Systems, pages 12593-12603, 2020.
An Xu and Heng Huang. Detached error feedback for distributed sgd with random sparsification. In International Conference on Machine Learning, pages 24550-24575. PMLR, 2022.
Junchi Yang, Xiang Li, Ilyas Fatkhullin, and Niao He. Two sides of one coin: the limits of untuned SGD and the power of adaptive methods. In Advances in Neural Information Processing Systems, 2023.
Tianbao Yang, Qihang Lin, and Zhe Li. Unified convergence analysis of stochastic momentum methods for convex and non-convex optimization. arXiv preprint arXiv:1604.03257, 2016.
Chung-Yiu Yau and Hoi-To Wai. DoCoM-SGT: Doubly compressed momentum-assisted stochastic gradient tracking algorithm for communication efficient decentralized learning. arXiv preprint arXiv:2202.00255, 2022.
Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference on Machine Learning, page 7184-7193. PMLR, 2019.
SK Zavriev and FV Kostyuk. Heavy-ball method in nonconvex optimization problems. Computational Mathematics and Modeling, 4(4):336-341, 1993.
Haoyu Zhao, Boyue Li, Zhize Li, Peter Richtárik, and Yuejie Chi. BEER: Fast O(1/T) rate for decentralized nonconvex optimization with communication compression. In Advances in Neural Information Processing Systems, 2022.
Shuai Zheng, Ziyue Huang, and James Kwok. Communication-efficient distributed blockwise momentum SGD with error-feedback. In Advances in Neural Information Processing Systems, 2019.
1 Introduction 1
2 Communication Compression, Error Feedback, and Sample Complexity 2
2.1 Brief history of error-feedback 2
2.2 Key issue: error feedback has an unhealthy appetite for samples! 3
2.3Mysterious effectiveness of momentum in nonconvex optimization 4
3 Main Results 5
3.1 A deeper dive into the issues EF21 has with stochastic gradients 6
3.2 Momentum for avoiding mega-batches 6
3.3 Distributed stochastic error feedback with momentum 7
3.4 Further improvement using double momentum! 9
4 Experiments 9
4.1 Experiment 1: increasing batch-size 10
4.2 Experiment 2: improving convergence with  $n$  10
A More on Contractive Compressors, Error Feedback and Momentum 18
B Variance Reduction Effect of SGDM and Comparison to STORM 21
C Additional Experiments and Details of Experimental Setup 22
C.1 Extra plots for experiments 1 and 2 22
C.2 Experiment 3: stochastic quadratic optimization 22
C.3 Experiment 4: training neural network 24
D Descent Lemma 25
E EF21-SGDM-ideal (Proof of Theorem 1 and Proposition 1) 25
F EF21-SGDM (Proof of Theorems 2 and 3) 28
F.1 Controlling the error of momentum estimator 30
F.2 Controlling the error of contractive compression and momentum estimator 31
G Further Improvement Using Double Momentum (Proof of Corollary 3) 33
G.1 Controlling the error of second momentum estimator 36
G.2 Controlling the error of contractive compression and double momentum estimator. 38
H EF21-SGDM with Absolute Compressor 40
H.1 Controlling the error of absolute compression 42
I EF21-STORM/MVR 43
I.1 Controlling the variance of STORM/MVR estimator 46
I.2 Controlling the variance of contractive compression and STORM/MVR estimator 47
J Simplified Proof of SGDM: Time Varying Parameters and No Tuning for Momentum Sequence 49
K Revisiting EF14-SGD Analysis under BG and BGS Assumptions 51
Method	Comm. compl.	Batch-size for comm. compl.	Asymp. sample compl.	Batch free?	No extra assum.?
EF14-SGD [Koloskova et al., 2020]	KGLmax/αε3	ασ2(*) / nεG	Lmaxσ2/nε4	✓	X (a)
NEOLITHIC [Huang et al., 2022]	KLmax/αε2 log (G/ε) (b)	σ2/nε2 ∨ 1/α log (G/ε) (*)	Lmaxσ2/nε4	X	X (c)
EF21-SGD [Fatkhullin et al., 2021]	KL/αε2	σ2/α2ε2	Lσ2/α3ε4 (d)	X	✓
BEER [Zhao et al., 2022]	KLmax/αε2	σ2/αε2	Lmaxσ2/d/α2ε4	X	✓
EF21-SGDM Corollary 2	KL/αε2	αL σ2/ L nε2 ∨ αL2/ L2 σ2/ ε2	Lσ2/nε4	✓	✓
EF21-SGD2M Corollary 3	KL/αε2	αL σ2/ L nε2 ∨ αL3/ L3 σ2/ ε2	Lσ2/nε4	✓	✓
(a) Analysis requires a bound of the second moment of the stochastic gradients, i.e.,  $\mathbb{E}\left[\| \nabla f_i(x,\xi_i)\|^2\right] \leq G^2$  for all  $x \in \mathbb{R}^d$ .
(b) This complexity is achieved by using a large mini-batch and communicating  $\left\lceil K / \alpha \right\rceil \approx d$  coordinates per iteration.
(c) Analysis requires a bounded gradient disimilarity assumption, i.e.,  $\frac{1}{n}\sum_{i = 1}^{n}\| \nabla f_i(x) - \nabla f(x)\| ^2\leq G^2$  for all  $x\in \mathbb{R}^d$
(d) Analysis requires a batch-size at least  $B \geq \frac{\sigma^2}{\sigma^2\varepsilon^2}$  for EF21-SGD and  $B \geq \frac{\sigma^2}{\sigma^2\varepsilon^2}$  for BEER.
(*) For a fair comparison, we take the (minimal) batch-size for these methods which guarantees the reported communication complexity.
Table 2: Extended summary of related works on distributed error compensated SGD methods using a Top  $K$  compressor under Assumptions 1 and 2. The goal is to find an  $\varepsilon$ -stationary point of a smooth nonconvex function of the form (1), i.e., a point  $x$  such that  $\mathbb{E}\left[\|\nabla f(x)\|\right] \leq \varepsilon$ . "Comm. compl." reports the total number of communicated bits if the method is applied with batch-size equal to "Batch-size" at each node. When Top  $K$  compressor is applied, then  $\alpha \geq K / d$ , and the comm. compl. of error compensated methods can be reduced by a factor of  $\alpha d / K$ . "Batch-size for comm. compl." means the batch-size for achieving the reported "Comm. compl." "Asymp. sample compl." reports the asymptotic sample complexity of the algorithm with batch-size  $B = 1$  in the regime  $\varepsilon \rightarrow 0$ , i.e., the total number of samples required at each node to achieve  $\varepsilon$ -stationary point. "Batch free" marks with  $\checkmark$  if the analysis ensures convergence with batch-size equal to 1. "No extra assum." marks with  $\checkmark$  if no additional assumption beyond Assumptions 1 and 2 is required for analysis. We denote  $L_{\max} := \max_{i \in [n]} L_i$ . Notice that it always holds  $L \leq \widetilde{L} \leq L_{\max}$  and these inequalities only become equalities in the homogeneous case. It could be that  $\alpha L / \widetilde{L} \ll 1$  making the batch-size of EF21-SGDM and EF21-SGD2M much smaller than those of EF21-SGD and BEER. Symbol  $\vee$  denotes the maximum of two scalars.
A More on Contractive Compressors, Error Feedback and Momentum
Greedy vs uniform. In our work, we specifically focus on the class of contractive compressors satisfying Definition 1, which contains a greedy Top  $K$  compressor as a special case. Note that Top  $K$  is greedy in that it minimizes the error  $\| \mathrm{Top}K(x) - x\|^2$  subject to the sparsity constraint  $\| \mathcal{C}(x)\|_0 \leq K$ , where  $\| u\|_0$  counts the number of nonzero entries in  $u$ . In practice, greediness is almost always very useful, translating into excellent empirical performance, especially when compared to the performance of the Rand  $K$  sparsifier. On the other hand, it appears to be very hard to formalize these practical gains theoretically. In fact, while greedy compressors such as Top  $K$  outperform their randomized cousins such as Rand  $K$  in practice, and often by huge margins [Lin et al., 2018], the theoretical picture is exactly reversed, and the theoretical communication complexity of gradient-type methods based on randomized compressors [Alistarh et al., 2017, Mishchenko et al., 2019, Horváth et al., 2019b, Li et al., 2020, Gorbunov et al., 2021] is much better than of those based on greedy compressors [Koloskova et al., 2020, Richtárik et al., 2021, Fatkhullin et al., 2021, Richtárik et al., 2022]. The key reason behind this is the fact that popular randomized compressors such as Rand  $K$  become unbiased mappings after appropriate scaling (e.g.,  $\mathbb{E}\left[\frac{d}{K}\mathrm{Rand}K(x)\right] \equiv x$ ), and that the inherent randomness is typically drawn independently by all clients. This leads to several key simplifications in the analysis, and consequently, to theoretical gains over methods that do not compress, and over methods that compress greedily. Further improvements are possible when the randomness is correlated in an appropriate way [Szlendak et al., 2021].
Due to the superior empirical properties of greedy contractive compressors, and our desire to push this very potent line of work further, in this paper we work with the general class of compressors
$^{17}$ Greediness is not useful, for example, when  $\mathcal{D}_i = \mathcal{D}_j$  for all  $i,j$  and when  $\mathrm{Top}K$  is applied to the full-batch gradient  $\nabla f_i(x)$  by each client. However, situations of this type arise rarely in practice.
$^{18}$ No theoretical results of this type exist for  $n > 1$
satisfying Definition 1, and do not invoke any additional restrictive assumptions. For example, we do not assume  $\mathcal{C}$  can be made unbiased after scaling.
Error Feedback. The first theoretical analysis of EF14 was presented in the works of Stich et al. [2018], Alistarh et al. [2018] and further revisited in convex case in [Karimireddy et al., 2019, Beznosikov et al., 2020, Gorbunov et al., 2020, Qian et al., 2020] and analysis was extended to nonconvex setting in [Stich and Karimireddy, 2019]. Later, in nonconvex case, various extensions and combinations of EF14 with other optimization techniques were considered and analyzed, which include bidirectional compression [Tang et al., 2020], decentralized training [Koloskova et al., 2020, Singh et al., 2021], server level momentum [Xie et al., 2020], client level momentum [Zheng et al., 2019], combination with adaptive methods [Li et al., 2022b]. To our knowledge, the best sample complexity for finding a stationary point for this method (including its momentum and adaptive variants) in the distributed nonconvex setting is given by Koloskova et al. [2020], which is  $\mathcal{O}\left(\frac{G}{\alpha\varepsilon^3} +\frac{\sigma^2}{n\varepsilon^4}\right)$ . More recently, Huang et al. [2022] propose a modification of EF14 method achieving  $\mathcal{O}\left(\frac{1}{\alpha\varepsilon^2}\log \left(\frac{G}{\varepsilon}\right) + \frac{\sigma^2}{n\varepsilon^4}\right)$  sample complexity by using the BGS assumption. When applied with Top  $K$  compressor, this method requires to communicate  $\widetilde{\mathcal{O}}\left(K / \alpha\right)$  coordinates at every iteration. This makes it impractical since when the effective  $\alpha$  is unknown and is set to  $\alpha = K / d$ , it means that their method communicates all  $d$  coordinates at every iteration, mimicking vanilla (S)GD method. Moreover, their algorithm uses an additional subroutine and applies updates with a large batch-size of samples of order  $\mathcal{O}\left(\frac{1}{\alpha}\log \left(\frac{G}{\varepsilon}\right)\right)$ , making the algorithm less practical and difficult to implement. It is worth to mention, that error feedback was also analyzed for other classes of compressors such as absolute (see Definition 2) or additive compressors (i.e.,  $\mathcal{C}(x + y) = \mathcal{C}(x) + \mathcal{C}(y)$  for all  $x,y\in \mathbb{R}^d$ ) [Tang et al., 2020, Xu and Huang, 2022], which do not include Top  $K$  sparsifier.
Momentum. The first convergence analysis of gradient descent with momentum was proposed by B.T. Polyak in his seminal work [Polyak, 1964] studying the benefit of multi-step methods. The proof technique proposed in this work is based on the analysis of the spectral norm of a certain matrix arising from the dynamics of a multi-step process on a quadratic function. Unfortunately, such technique is restricted to the case of strongly convex quadratic objective and the setting of full gradients. Later Zavriev and Kostyuk [1993] prove an asymptotic convergence of this method in nonconvex deterministic case without specifying the rate of convergence.
To our knowledge, the first non-asymptotic analysis of SGDM in the smooth nonconvex setting is due to Yu et al. [2019]. Their analysis, however, heavily relies on BG assumption. Later, Liu et al. [2020] provide a refined analysis of SGDM, removing the BG assumption and improving the dependence on the momentum parameter  $\eta$ . Notice that the analysis of Liu et al. [2020] and the majority of other works relies on some variant of the following Lyapunov function:
$$
\Lambda_ {t} := f \left(z ^ {t}\right) - f ^ {*} + \sum_ {\tau = 0} ^ {t} c _ {\tau} \left\| x ^ {t - \tau} - x ^ {t - 1 - \tau} \right\| ^ {2}, \tag {11}
$$
where  $\{z^t\}_{t\geq 0}$  is some auxiliary sequence (often) different from the iterates  $\{x^{t}\}_{t\geq 0}$ , and  $\{c_{\tau}\}_{\tau \geq 0}$  is a diminishing non-negative sequence. This approach is motivated by the dynamical system point of view at Polyak's heavy ball momentum, where the two terms in (11) are interpreted as the potential and kinetic energy of the system [Sebbouh et al., 2019]. In contrast, the Lyapunov function used in this work is conceptually different even in the single node  $(n = 1)$ , uncompressed  $(\alpha = 1)$  setting. Later, Defazio [2021] revisit the analysis in [Liu et al., 2020] through the lens of primal averaging and provide insights on why momentum helps in practice. The momentum is also used for stabilizing adaptive algorithms such as normalized SGD [Cutkosky and Mehta, 2020]. In particular, it was shown that by using momentum, one can ensure convergence without large batches for normalized SGD (while keeping the same sample complexity as a large batch normalized SGD). However, their analysis is specific to the normalized method, which allows using the function value as a Lyapunov function. High probability analysis of momentum methods was investigated in [Cutkosky and Mehta, 2021, Li and Orabona, 2020]. In the distributed setting, [Yu et al., 2019, Karimireddy et al., 2021] extend the analysis of SGDM under BGS assumption. Later [Takezawa et al., 2022, Gao et al., 2023] remove this assumption providing a refined analysis based on the techniques developed in [Liu et al., 2020]. However, the algorithms in these works do not apply any bandwidth reduction technique such as communication compression.
We would like to mention that understanding the behavior of SGDM in convex case also remains an active area of research [Ghadimi et al., 2015, Yang et al., 2016, Sebbouh et al., 2021, Li et al., 2022a, Xiao and Yang, 2022].
B Variance Reduction Effect of SGDM and Comparison to STORM
Notice that the choice of our Lyapunov function  $\Lambda_t$  (8), which is used in the analysis of EF21-SGDM implies that the gradient estimators  $g_i^t$  and  $v_i^t$  improve over the iterations, i.e.,
$$
g _ {i} ^ {t} \rightarrow \nabla f _ {i} \left(x ^ {t}\right), \quad v _ {i} ^ {t} \rightarrow \nabla f _ {i} \left(x ^ {t}\right) \quad \text {f o r} t \rightarrow \infty .
$$
This comes in contrast with the behavior of SGD, for which the gradient estimator  $v_{i}^{t} = \nabla f_{i}(x^{t},\xi_{i}^{t})$  does not necessarily tend to zero over iterations. Such effect of asymptotic improvement of the estimation error of the gradient estimator is reminiscent to the analogous effect known in the literature on variance reduction (VR) methods. In particular, the classical momentum step 6 of Algorithm 1 may be contrasted with a STORM variance reduced estimator proposed by Cutkosky and Orabona [2019], which updates the gradient estimator via
$$
w _ {i} ^ {t + 1} = \nabla f _ {i} \left(x ^ {t + 1}, \xi_ {i} ^ {t + 1}\right) + (1 - \eta) \left(w _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t + 1}\right)\right), \quad w _ {i} ^ {0} = \nabla f _ {i} \left(x ^ {0}, \xi_ {i} ^ {0}\right) \tag {12}
$$
It is known that the class of VR methods (and STORM, in particular) can show faster asymptotic convergence in terms of  $T$  (or  $\varepsilon$ ) compared to SGD and SGDM under some additional assumptions. However, we would like to point out the important differences (and limitations) of (12) compared to the classical Polyak's momentum used on line 6 of Algorithm 1. First, the estimator  $w_{i}^{t + 1}$  is different from the momentum update rule  $v_{i}^{t + 1}$  in that it is unbiased for any  $t \geq 0$ , i.e.,  $\mathbb{E}\left[w_i^{t + 1} - \nabla f_i(x^{t + 1})\right] = 0$ , which greatly facilitates the analysis of this method. Notice that, in particular, in the deterministic case ( $\sigma = 0$ ,  $\alpha = 1$ ), the method with estimator (12) reduces to vanilla gradient descent with  $w_{i}^{t + 1} = \nabla f_{i}(x^{t + 1})$ . Second, the computation of  $w_{i}^{t + 1}$  requires access to two stochastic gradients  $\nabla f_{i}(x^{t + 1},\xi_{i}^{t + 1})$  and  $\nabla f_{i}(x^{t},\xi_{i}^{t + 1})$  under the same realization of noise  $\xi_{i}^{t + 1}$  at each iteration, and requires the additional storage of control variate  $x^t$ . This is a serious limitation, which can make the method impractical or even not implementable for certain applications such as federated RL [Mitra et al., 2023], multi-agent RL [Doan et al., 2019] or operations research problems [Chen et al., 2022]. Third, the analysis of variance reduced methods such as STORM requires an additional assumptions such as individual smoothness of stochastic functions (or its averaged variants) (Assumption 3), i.e.,  $\| \nabla f_i(x,\xi_i) - \nabla f_i(y,\xi_i)\| \leq \ell_i\| x - y\|$  for all  $x,y \in \mathbb{R}^d$ ,  $\xi_{i} \sim \mathcal{D}_{i}$ ,  $i \in [n]$ , while our EF21-SGDM only needs smoothness of (deterministic) local functions  $f_{i}(x)$ . While this assumption is satisfied for some loss functions in supervised learning, it can also be very limiting. Even if Assumption 3 is satisfied, the constant  $\widetilde{\ell}$  (which always satisfies  $\widetilde{\ell} \geq \widetilde{L}$ ) can be much larger than  $\widetilde{L}$  canceling the speed-up in terms of  $T$  (or  $\varepsilon$ ). For completeness, we provide the sample complexity analysis of our error compensated method combined with estimator (12), which is deferred to Appendix I.
19Notice that  $\mathbb{E}\left[w_i^0 -\nabla f_i(x^0)\right] = 0$  . Let  $\mathbb{E}\left[w_i^t -\nabla f_i(x^t)\right] = 0$  hold, then
$\mathbb{E}\left[w_i^{t + 1} - \nabla f_i(x^{t + 1})\right] = (1 - \eta)\mathbb{E}\left[w_i^t -\nabla f_i(x^t,\xi_i^{t + 1})\right] = 0.$
(a) Divergence in single node setting,  $n = 1$ .
(b) No improvement with  $n$
Figure 4: Divergence of EF21-SGD on a quadratic function  $\frac{1}{2} \|x\|^2$  with Top1 compressor. See the proof of Theorem 1 for details on the construction of noise  $\xi$ , we use  $\sigma = 1$ ,  $B = 1$ . The starting point for all methods is  $x^0 = (0, -0.01)^\top$ . Unlike Figure 1, these experiments use time varying step-sizes and momentum parameters  $\gamma_t = \eta_t = \frac{0.1}{\sqrt{t + 1}}$ . Each method is run 10 times and the plot shows the median performance alongside the  $25\%$  and  $75\%$  quantiles.
C Additional Experiments and Details of Experimental Setup
Divergence of EF21-SGD with time-varying step-sizes. We complement our Figure 1 in the main part of the paper, which shows divergence of EF21-SGD [Fatkhullin et al., 2021] with small (constant) step-size. Here, in Figure 4, we see that the similar divergence is observed when using time varying step-sizes  $\gamma_{t} = \frac{0.1}{\sqrt{t + 1}}$ . Also, EF21-SGD with time-varying step-size does not improve convergence when  $n$  is increased.
Implementation Details. The experiments were implemented in Python 3.7.9. The distributed environment was emulated on machines with Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz. In all experiments with MNIST, we split the dataset across nodes by labels to simulate the heterogeneous setting.
C.1 Extra plots for experiments 1 and 2
In Figures 5 and 6, we provide extra experiments for the setup from Section 4.
(b)  $B = 32$
Figure 5: Performance of algorithms on MNIST dataset with  $n = 100$ , and Top10 compressor.
C.2 Experiment 3: stochastic quadratic optimization
We now consider a synthetic  $\lambda$ -strongly convex quadratic function problem  $f(x) = \frac{1}{n}\sum_{i=1}^{n}f_{i}(x)$ , where the functions  $f_{i}(x) = \frac{1}{2}x^{\top}\mathbf{Q}_{i}x - x^{\top}b_{i}$  are (not necessarily convex) quadratic functions for all  $i \in [n]$  and  $x \in \mathbb{R}^d$ . The matrices  $\mathbf{Q}_1, \dots, \mathbf{Q}_n$ , vectors  $b_1, \dots, b_n$ , and a starting point  $x^0$  are generated by Algorithm 2 with the number of nodes  $n = 100$ , dimension  $d = 1000$ , regularizer  $\lambda = 0.01$ , and scale  $s = 1$ . For all  $i \in [n]$  and  $x \in \mathbb{R}^d$ , we consider stochastic gradients  $\nabla f_{i}(x,\xi) = \nabla f_{i}(x) + \xi_{i}$ , where  $\xi_{i}$  are i.i.d. samples from  $\mathcal{N}(0,\sigma)$  with  $\sigma \in \{0.001, 0.01\}$ . In
(a)  $n = 1$
(b)  $n = 10$
Figure 6: Performance of algorithms on real-sim dataset with batch-size  $B = 1$ , and Top100 compressor.
Figure 7, we present the comparison of EF21-SGDM and EF14-SGD with three different step sizes. The behavior of methods for other step sizes from the set  $\{2^k \mid k \in [-20, 20]\}$  follows a similar trend. For every step size, we observe that at the beginning, the methods have almost the same linear rates, but then EF14-SGD gets stuck at high accuracies, while EF21-SGDM continues converging to the lower accuracies.
Figure 7: Stochastic Quadratic Optimization Problem with  $\sigma = 0.001$  (left figure) and  $\sigma = 0.01$  (right figure)
Algorithm 2 Quadratic Optimization Task Generation Procedure
1: Parameters: number nodes  $n$ , dimension  $d$ , regularizer  $\lambda$ , and scale  $s$ .
2: for  $i = 1, \dots, n$  do
3: Calculate Gaussian noises  $\mu_i^s = 1 + s\xi_i^s$  and  $\mu_i^b = s\xi_i^b$ , i.i.d.  $\xi_i^s, \xi_i^b \sim \mathcal{N}(0,1)$
4:  $b_{i} = \frac{\mu_{i}^{s}}{\lambda} (-1 + \mu_{i}^{b}, 0, \dots, 0) \in \mathbb{R}^{d}$
5: Scale the predefined tridiagonal matrix
$$
\mathbf {Q} _ {i} = \frac {\mu_ {i} ^ {s}}{4} \left( \begin{array}{c c c c} 2 & - 1 & & 0 \\ - 1 & \ddots & \ddots & \\ & \ddots & \ddots & - 1 \\ 0 & & - 1 & 2 \end{array} \right) \in \mathbb {R} ^ {d \times d}
$$
6: end for
7: Find the mean of matrices  $\mathbf{Q} = \frac{1}{n}\sum_{i = 1}^{n}\mathbf{Q}_i$
8: Find the minimum eigenvalue  $\lambda_{\mathrm{min}}^{\prime}(\mathbf{Q})$
9: for  $i = 1,\dots ,n$  do
10:Normalize matrix  $\mathbf{Q}_i = \mathbf{Q}_i + (\lambda -\lambda_{\min}(\mathbf{Q}))\mathbf{I}$
1: end for
2: Find a starting point  $x^0 = (\sqrt{d}, 0, \dots, 0)$ .
13: Output a new problem: matrices  $\mathbf{Q}_1, \dots, \mathbf{Q}_n$ , vectors  $b_1, \dots, b_n$ , starting point  $x^0$