SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Pullback, a dynamic stepsize  $\eta_t = \eta / \| v_t\|$  in the descent phase where  $\eta = O(\epsilon)$  and  $v_{t}$  is the gradient estimator. This stepsize is originated from SPIDER (Fang et al. [2018]) which ensures its convergence by bounding the update distance  $\| x_{t + 1} - x_t\|$ . In the escaping phase, Pullback adopts a larger stepsize of  $O(1)$  in the escaping phase and a special pullback stepsize in the last iteration, which is the key to improve the gradient complexity. Different from Pullback, in Algorithm 1 we adopt a consistent stepsize such that it keeps invariant even if phase changes and all nodes always use the same stepsize. If there is no perturbation in iteration  $t$ , we have  $\bar{x}_{t + 1} = \bar{x}_t - \eta \bar{v}_t$  which is important to the convergence analysis. We discard the strategy in Pullback for two reasons. First, the gradient normalization will probably cause divergence issues in decentralized optimization because in centralized algorithm the gradient direction is  $v_{t} / \| v_{t}\|$ , which is equivalent to  $v_{c} = \sum_{i = 1}^{n}v_{t}^{(i)} / \| \sum_{i = 1}^{n}v_{t}^{(i)}\|$ . However, in decentralized algorithm the average of  $v_{t}^{(i)}$  is not available on local nodes. If the gradient normalization is done locally, we will get  $v_{d} = \sum_{i = 1}^{n}v_{t}^{(i)} / \| v_{t}^{(i)}\|$ , which is different to  $v_{c}$  and the error is hard to be estimated. Actually, both D-GET and D-SPIDER-SFO adopt the constant stepsize in SpiderBoost (Wang et al. [2019]) to avoid performing the gradient normalization step. SPIDER needs the gradient normalization because  $\| x_{t + 1} - x_t\|$  is required to be small in the proof, while SpiderBoost improves the proof to bound  $\| x_{t + 1} - x_t\|$  by  $\eta \| v_t\|$  which is canceled eventually. In our analysis we also adopt the strategy in SpiderBoost. Second, in our algorithm the changing of phase is occurred independently on each node. The phase-wise stepsize and pullback strategy will lead to different stepsizes among all nodes in one iteration, which will also cause potential convergence issues.
In (Chen et al. [2022]), two versions of Pullback are proposed, i.e., Pullback-SPIDER and Pullback-STORM using SPIDER and STORM as the gradient estimator respectively. As introduced previously, one of the advantages of STORM is avoiding large batchsize. Nonetheless, Pullback-STORM adopt a large batchsize of  $\tilde{O}(\epsilon^{-1})$  in each iteration, which violates the original intention of STORM. Besides, from Table 1 we can see all algorithms achieving second-order optimality with  $\tilde{O}(\epsilon^{-3})$  gradient complexity require a large batchsize of  $O(\epsilon^{-1})$ . Therefore, we propose a small batch version named PEDESTAL-S as a special case of PEDESTAL that only requires an averaged batchsize of  $O(1)$ .
3.2.3 Conditions of Termination
As a result of applying gradient tracking, we can bound  $\frac{1}{n}\sum_{i=1}^{n}\|y_t^{(i)} - \bar{y}_t\|^2$  by  $O(\epsilon^2)$ . Even though we have such an estimation, it is still possible that the norm  $\|y_t^{(i)}\|$  is as large as  $O(\sqrt{n}\epsilon)$  on some nodes when the entire decentralized network has already achieved optimality. Therefore, waiting for all nodes to reach second-order stationary point is not an efficient strategy. This is the reason why we terminate our algorithm when only a fraction of worker nodes satisfy  $esc^{(i)} \geq C_T$ .
In SSRGD and Pullback, there is an upper bound of iteration numbers in the escaping phase. If the escaping phase is not broken in this number of iterations then the candidate point is regarded as a second-order stationary point. If the escaping phase is broken, then the averaged moving distance is larger than a threshold and the loss function will be reduced by  $O(\epsilon^2)$  on average. This strategy guarantees that the algorithm will terminate with a certain gradient complexity. However, in our algorithm worker nodes do not enter escaping phase simultaneously and thus we do not set such an upper bound. In this case the averaged moving distance cannot be lower bounded as  $C_T$  has no upper bound. Fortunately, we can complete our analysis by a different novel framework (see the proof outline in the Appendix). An alternative solution is to stop the update on the node that has run certain number of iterations in the escaping phase while the algorithm will continue. But that solution is also challenging since the relation between the first-achieved local optimal solution and the final global optimal solution is unknown and the analysis is non-trivial.
One remaining issue of the current termination strategy is that it involves the global knowledge of how many worker nodes satisfying the termination condition. One solution is to run an additional process to track this global value. The cost of transmitting Boolean values is much less expensive than broadcasting the model. Another solution is to set a maximum iteration in practice. Generally we need to evaluate the model after certain epochs to see if the training process is running smoothly and we can save a checkpoint when we find a better evaluation result. The theoretical analysis ensures that an optimal solution can be visited if the number of iterations is as large as  $O(\epsilon^{-3})$ .
3.2.4 Small Stuck Region
The theoretical guarantee of second-order optimality in SSRGD and Pullback is mainly credit to the lemma of small stuck region, which states that if there are two decoupled sequences  $x_{t}$  and  $x_{t}^{\prime}$  with identical stochastic samples,  $x_{s} = x_{s}^{\prime}$  and  $x_{s+1} - x_{s+1}^{\prime} = r_{0}\mathbf{e}_{1}$  where  $\mathbf{e}_{1}$  is the eigenvector corresponding to the smallest eigenvalue, then it satisfies  $\max\{\|x_{t} - x_{s}\|, \|x_{t}^{\prime} - x_{s}^{\prime}\| \geq C_{d}$  for some  $s \leq t \leq s + C_{T}$  with high probability. In SSRGD and Pullback, the averaged moving distance  $\frac{1}{t-s}\sum_{\tau=s+1}^{t}\|x_{\tau+1} - x_{\tau}\|^{2}$  is used as the criterion to discriminate saddle point because the small stuck region lemma can be applied in this way. However, in decentralized algorithm some nodes enter the escaping phase before the candidate point  $\bar{x}_{s}$  is achieved. Suppose node  $i$  enters escaping phase in iteration  $s^{\prime}$ , then the averaged moving distance starting from iteration  $s$  on node  $i$  cannot be well estimated because the condition of not breaking escaping phase on node  $i$  only guarantees the bound of averaged moving distance starting from  $s^{\prime}$ . Therefore, in our method we use the total moving distance  $\|x_{t}^{(i)} - x_{s}^{(i)}\|$  as the criterion because we can obtain estimation  $\|x_{t}^{(i)} - x_{s}^{(i)}\| \leq 2C_{d}$  given  $\|x_{t}^{(i)} - x_{s^{\prime}}^{(i)}\| \leq C_{d}$  and  $\|x_{s}^{(i)} - x_{s^{\prime}}^{(i)}\| \leq C_{d}$ . And we can further complete our analysis by the small stuck region lemma. Actually we do not require more memory because  $\tilde{x}$  is the point to return in SSRGD and Pullback (hence should be saved). In practice, we can also return  $\tilde{x}^{(i)}$  for any node  $i$  drawing perturbation in iteration  $t - C_{T}$  since  $\|x_{t}^{(i)} - \bar{x}_{t}\|$  can be well bounded. Besides, we discover that the consensus error  $\frac{1}{n}\sum_{i=1}^{n}\|x_{t}^{(i)} - \bar{x}_{t}\|^{2}$  results in an extra term when proving the small stuck region lemma, which becomes another challenge. If the consensus error is not under control, it can drive  $x$  away from  $x_{s}$  or push  $x$  toward  $x_{s}$ , no matter what  $\nabla f(x)$  is. In this manner, the stuck region cannot be estimated. In this work, we provide the corresponding proof to estimate this new term exclusively occurred in decentralized setting in our convergence analysis.
In this section we will provide the main theorem of our convergence analysis. First we will introduce the assumptions used in this paper. All assumptions used in this paper are mild and commonly used in the analysis of related works.
Assumption 1. (Lower Bound) The objective  $f$  is lower bounded, i.e.,  $\inf_{x} f(x) = f^{*} > -\infty$ .
Assumption 2. (Bounded Variance) The stochastic gradient of each local loss function is an unbiased estimator and has bounded variance, i.e., for any  $i \in \{1, 2, \dots, n\}$  we have
$$
\mathbb {E} _ {\xi} \nabla F _ {i} (x, \xi) = \nabla f _ {i} (x), \mathbb {E} _ {\xi} \| \nabla F _ {i} (x, \xi) - \nabla f _ {i} (x) \| ^ {2} \leq \sigma^ {2} \tag {2}
$$
Assumption 3. (Lipschitz Gradient) For all  $\xi$  and  $i\in \{1,2,\dots ,n\}$ ,  $F_{i}(x,\xi)$  has Lipschitz gradient, i.e., for any  $x_{1}$  and  $x_{2}$  we have  $\| \nabla F_i(x_1,\xi) - \nabla F_i(x_2,\xi)\| \leq L\| x_1 - x_2\|$  with a constant  $L$ .
Assumption 4. (Lipschitz Hessian) For all  $\xi$  and  $i\in \{1,2,\dots ,n\}$ ,  $F_{i}(x,\xi)$  has Lipschitz hessian, i.e., for any  $x_{1}$  and  $x_{2}$  we have  $\| \nabla^{2}F_{i}(x_{1},\xi) - \nabla^{2}F_{i}(x_{2},\xi)\| \leq \rho \| x_{1} - x_{2}\|$  with a constant  $\rho$ .
Assumption 1, Assumption 2 and Assumption 3 are common assumptions used in the analysis of stochastic optimization algorithms. Assumption 4 is the standard assumption to find second-order optimality, which is used in all algorithms that achieves second-order stationary point in Table 1.
Assumption 5. (Spectral Gap) The decentralized network is connected by a doubly-stochastic weight matrix  $W \in \mathbb{R}^{n \times n}$  satisfying  $W\mathbf{1}_n = W^T\mathbf{1}_n = \mathbf{1}_n$  and  $\lambda \coloneqq \| W - J\| \in [0,1)$ .
Here  $J$  is a  $n \times n$  matrix with all elements equal to  $\frac{1}{n}$ .  $W$  is the weight matrix of the decentralized network where  $w_{ij} > 0$  if node  $i$  and node  $j$  are connected, otherwise  $w_{ij} = 0$ .  $\| \cdot \|$  denotes the spectral norm of matrix (i.e., largest singular value). Notice that  $\lambda$  is a connectivity measurement of the network graph and it is also the second largest singular value of  $W$ . We do not assume  $W$  to be symmetric and hence the communication network can be both directed graph and undirected graph. The spectral gap assumption is also used commonly in the analysis of decentralized algorithms.
4.2 Main Theorems
Let  $\epsilon_H = \epsilon^\alpha$ . When  $\alpha \leq 0.5$ , we have the following Theorem 1.
Theorem 1. Assume  $\alpha \leq 0.5$  and Assumption 1 to 5 are satisfied. Let  $\theta = \min \left\{\frac{3 - 5\alpha}{2}, 1\right\}$ . We set  $\eta = \Theta\left(\frac{\epsilon^{\theta}}{L}\right)$ ,  $\beta = \Theta(\epsilon^{1 + \theta})$ ,  $b_{0} = \Theta(\epsilon^{-2})$ ,  $b_{1} = \Theta(\max\{\epsilon^{2 - \theta - 5\alpha}, 1\})$ ,  $r = \Theta(\epsilon^{1 + \theta})$ ,  $C_{v} = \Theta(\epsilon)$ ,  $C_{T} = \tilde{\Theta}(\epsilon^{-\theta - \alpha})$  and  $C_{d} = \tilde{\Theta}(\epsilon^{1 - \alpha})$ . Then our PEDESTAL algorithm will achieve  $O(\epsilon, \epsilon_{H})$ -second-order stationary point with  $\tilde{O}(\epsilon^{-3})$  gradient complexity.
The specific constants hidden in  $\Theta (\cdot)$  will be shown in Appendix B, where the proof outline and the completed proof of Theorem 1 can also be found. From Theorem 1 we can see our PEDESTAL-S with  $b_{1} = O(1)$  can achieve  $O(\epsilon ,\epsilon_{H})$  -second-order stationary point with  $\tilde{O} (\epsilon^{-3})$  gradient complexity for  $\epsilon_H\geq \epsilon^{0.2}$ . In the classic setting, our PEDESTAL achieves second-order stationary point with  $\tilde{O} (\epsilon^{-3})$  gradient complexity. When  $\alpha >0.5$ , i.e.,  $\epsilon_{H} < \sqrt{\epsilon}$ , we have the following Theorem 2. Since the parameter settings are different and the  $O(1)$  batchsize is only available in Theorem 1, we separate these two theorems. The proof of Theorem 2 can be found in Appendix D.
Theorem 2. When  $\epsilon_H < \sqrt{\epsilon}$  (i.e.,  $\alpha > 0.5$ ), we set  $\eta = \tilde{\Theta}(\epsilon^\theta)$ ,  $\beta = \Theta(\epsilon^{1+\theta})$ ,  $b_0 = \Theta(\epsilon^{-1})$ ,  $b_1 = \tilde{\Theta}(\epsilon^{-\max\{4\alpha - 1 - \theta, \theta + \alpha\}})$ ,  $r = \Theta(\epsilon^{1+\theta})$ ,  $C_v = \Theta(\epsilon)$ ,  $C_T = \tilde{\Theta}(\epsilon^{-\theta - \alpha})$  and  $C_d = \tilde{\Theta}(\epsilon^\alpha)$  where  $\theta = \min\{\frac{3\alpha - 1}{2}, 3\alpha - 2\}$ . Under Assumption 1 to 5, our PEDESTAL algorithm will achieve  $O(\epsilon, \epsilon_H)$ -second-order stationary point with  $\tilde{O}(\epsilon \epsilon_H^{-8} + \epsilon^4 \epsilon_H^{-11})$  gradient complexity.
In this section we will demonstrate our experimental results to validate the performance of our method. We conduct two tasks in our experiment, a matrix sensing task on synthetic dataset and a matrix factorization task on real-world dataset. Both of these two tasks are non-spurious local minimum problems (Ge et al. [2017, 2016]), which means all local minima are global minima. Thus, we conclude an algorithm is stuck at saddle point if the loss function value does not achieve the global minimum. The source code is available in https://github.com/WH-XIAN/PEDESTAL.
5.1 Matrix Sensing
We follow the experimental setup of (Chen et al. [2022]) to solve a decentralized matrix sensing problem. The goal of this task is to recover a low-rank  $d \times d$  symmetric matrix  $M^{*} = U^{*}(U^{*})^{T}$  where  $U^{*} \in \mathbb{R}^{d \times r}$  for some small  $r$ . We set the number of worker nodes to  $n = 20$ . We generate a synthetic dataset with  $N$  sensing matrices  $\{A_{i}\}_{i=1}^{N}$  and  $N$  corresponding observations  $b_{i} = \langle A_{i}, M^{*} \rangle$ . Here the inner product  $\langle X, Y \rangle$  of two matrices  $X$  and  $Y$  is defined by the trace  $tr(X^T Y)$ . The decentralized optimization problem can be formulated by
$$
\min  _ {U \in \mathbb {R} ^ {d \times r}} \sum_ {i = 1} ^ {n} L _ {i} (U), \text {w h e r e} L _ {i} (U) = \frac {1}{2} \sum_ {j = 1} ^ {N _ {i}} \left(\left\langle A _ {i j}, U U ^ {T} \right\rangle - b _ {i j}\right) ^ {2}, \tag {3}
$$
(a)  $d = 50$  ,ring
(b)  $d = 50$ , toroidal
(c)  $d = 50$  , exponential
(d)  $d = 100$  , ring
(e)  $d = 100$  toroidal
(f)  $d = 100$  , exponential
Figure 1: Experimental results of the decentralized matrix sensing task on different network topology for  $d = 50$  and  $d = 100$ . Data is assigned to worker nodes by random distribution. The y-axis is the loss function value and the x-axis is the number of gradient oracles divided by the number of data  $N$ .
(b) toroidal, random
(a) ring, random
(d) ring, Dirichlet
(e) toroidal, Dirichlet
Figure 2: Experimental results of the decentralized matrix factorization task on different network topology on MovieLens-100k. The y-axis is the loss function value and the x-axis is the number of gradient oracles divided by the size of matrix  $N \times l$ .
(c) exponential, random
(f) exponential, Dirichlet
where  $N_{i}$  is the amount of data assigned to worker node  $i$ .
The number of rows of matrix  $U$  is set to  $d = 50$  and  $d = 100$  respectively and the number of columns is set to  $r = 3$ . The ground truth low-rank matrix  $M^{*}$  equals  $U^{*}(U^{*})^{T}$  where each entry of  $U^{*}$  is generated by Gaussian distribution  $\mathcal{N}(0,1 / d)$  independently. We randomly generate  $N = 20\times n\times d$  samples of sensing matrices  $\{A_i\}_{i = 1}^N$ ,  $A_{i}\in \mathbb{R}^{d\times d}$  from standard Gaussian distribution and calculate the corresponding labels  $b_{i} = \langle \bar{A}_{i},M^{*}\rangle$ . We consider two different types of data distribution, random distribution and Dirichlet distribution  $Dir_{20}(0.3)$  to assign data to each worker node. We conduct experiments on three different types of network topology, i.e., ring topology, toroidal
topology (2-dimensional ring) and undirected exponential graph. The initial value of  $U$  is set to  $[u_0, \mathbf{0}, \mathbf{0}]$  where  $u_0$  is yield from Gaussian distribution and multiplied by a scalar such that it satisfies  $\| u_0 \| \leq \max eig(M^*)$ . We compare our PEDESTAL algorithm to decentralized baselines including D-PSGD, GTDSGD, D-GET, D-SPIDER-SFO and GTHSGD. In this experiment, the learning rate is chosen from  $\{0.01, 0.001, 0.0001\}$ . The batchsize is set to 10. For PEDESTAL and GTHSGD, parameter  $\beta$  is set to 0.01. For D-GET and D-SPIDER-SFO, the period  $q$  is 100. For PEDESTAL, threshold  $C_v$  is set to 0.0001. Perturbation radius  $r$  is set to 0.001. The threshold of moving distance  $C_d$  is set to 0.01. The experimental results are shown in Figure 1. Due to the space limit, we only show the result of random data distribution in the main manuscript and leave the result of Dirichlet distribution to Appendix A.
From the experimental result we can see all baselines are stuck at the saddle point and cannot escape it effectively. In contrast, our PEDESTAL reaches and escapes saddle points and finally find the local minimum. We also calculate the smallest eigenvalue of hessian matrix for each algorithm at the converged optimal point, which is left to the Supplementary Material because of space limit. According to the eigenvalue result, we can see the smallest eigenvalue is much closer to 0 than all baselines. Therefore, our experiment verifies that our PEDESTAL achieves the best performance to escape saddle point and find local minimum.
5.2 Matrix Factorization
The second task in our experiment is matrix factorization, which aims to approximate a given matrix  $M \in \mathbb{R}^{N \times l}$  by a low-rank matrix that can be decomposed to the product of two matrices  $U \in \mathbb{R}^{N \times r}$  and  $V \in \mathbb{R}^{l \times r}$  for some small  $r$ . The optimization problem can be formulated by
$$
\min  _ {U \in \mathbb {R} ^ {N \times r}, V \in \mathbb {R} ^ {l \times r}} \| M - U V ^ {T} \| _ {F} ^ {2} := \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {l} \left(M _ {i j} - \left(U V ^ {T}\right) _ {i j}\right) ^ {2} \tag {4}
$$
where  $\| \cdot \|_F$  denotes the Frobenius norm and subscript  $ij$  refers to the element at  $i$ -th row and  $j$ -th column. In our experiment we solve this problem on the MovieLens-100k dataset (Harper and Konstan [2015]). MovieLens-100k contains 100,000 ratings of 1682 movies provided by 943 users. Each rating is in the interval  $[0, 5]$  and scaled to  $[0, 1]$  in the experiment. This task can be regarded as an association task to predict users' potential ratings for unseen movies. In our experiment we set the number of worker nodes to  $n = 50$ . Each node is assigned the data from different group of users. Similar to the matrix sensing task, here we also use random distribution and Dirichlet distribution respectively to distribute users to worker nodes. And we also use ring topology, toroidal topology and undirected exponential graph as the communication network. The baselines are also D-PSGD, GTDSGD, D-GET, D-SPIDER-SFO and GTHSGD. In this experiment, the number of worker nodes is 50 and the rank of the matrix  $M$  is set to 25. The learning rate is chosen from  $\{0.01, 0.001, 0.0001\}$ . The batchsize is set to 100. For PEDESTAL and GTHSGD, parameter  $\beta$  is set to 0.1. For D-GET and D-SPIDER-SFO, the period  $q$  is 100. For PEDESTAL, threshold  $C_v$  is set to 0.002. Perturbation radius  $r$  is set to 0.01. The threshold of moving distance  $C_d$  is set to 0.5. The experimental results are shown in Figure 2.
From the experimental results we can see our PEDESTAL achieves the best performance to escape saddle point and find second-order stationary point. All baselines cannot escape saddle point effectively or efficiently. Particularly, variance reduced methods D-GET and D-SPIDER-SFO shows worse performance than SGD based algorithms D-PSGD and GTDSGD, which indicates that although reducing gradient noise can accelerate convergence, it also weakens the ability to escape saddle point. Therefore, our contribution is important since we make the fast convergence of variance reduction compatible with the capability to avoid saddle point.
In this paper we propose a novel algorithm PEDESTAL to find local minima in nonconvex decentralized optimization. PEDESTAL is the first decentralized stochastic algorithm to achieve second-order optimality with non-asymptotic analysis. We improve the drawbacks in previous deterministic counterpart to make phase changed independently on each node and avoid consensus protocols of broadcast or aggregation. We prove that PEDESTAL can achieve  $O(\epsilon, \sqrt{\epsilon})$ -second-order stationary point with the gradient complexity of  $\tilde{O}(\epsilon^{-3})$ , which matches state-of-the-art results of centralized counterpart or decentralized method to find first-order stationary point. We also conduct the matrix sensing and matrix factorization tasks in our experiments to validate the performance of PEDESTAL.
Taoxing Pan, Jun Liu, and Jie Wang. D-spider-sfo: A decentralized optimization algorithm with faster convergence rate for nonconvex problems. Proceedings of the AAAI Conference on Artificial Intelligence, 34(02):1619-1626, Apr. 2020. doi: 10.1609/aaai.v34i02.5523.
Haoran Sun, Songtao Lu, and Mingyi Hong. Improving the sample and communication complexity for decentralized non-convex optimization: Joint gradient estimation and tracking. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9217-9228. PMLR, 13-18 Jul 2020.
Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu.  $d^2$ : Decentralized training over decentralized data. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4848-4856. PMLR, 10-15 Jul 2018.
Isidoros Tziotis, Constantine Caramanis, and Aryan Mokhtari. Second order optimality in decentralized non-convex optimization via perturbed gradient tracking. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21162-21173. Curran Associates, Inc., 2020.
Stefan Vlaski and Ali H Sayed. Second-order guarantees in centralized, federated and decentralized nonconvex optimization. arXiv preprint arXiv:2003.14366, 2020.
Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, and Vahid Tarokh. Spiderboost and momentum: Faster variance reduction algorithms. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ran Xin, Usman Khan, and Soummya Kar. A hybrid variance-reduced method for decentralized stochastic non-convex optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11459–11469. PMLR, 18–24 Jul 2021a.
Ran Xin, Usman A. Khan, and Soummya Kar. An improved convergence analysis for decentralized online stochastic non-convex optimization. IEEE Transactions on Signal Processing, 69:1842-1858, 2021b. doi: 10.1109/TSP.2021.3062553.
Jinming Xu, Shanying Zhu, Yeng Chai Soh, and Lihua Xie. Augmented distributed gradient methods for multi-agent optimization under uncoordinated constant step sizes. In 2015 54th IEEE Conference on Decision and Control (CDC), pages 2055-2060, 2015. doi: 10.1109/CDC.2015.7402509.
Yi Xu, Rong Jin, and Tianbao Yang. First-order stochastic algorithms for escaping from saddle points in almost linear time. Advances in neural information processing systems, 31, 2018.
A Additional Experimental Results
The experimental results of Dirichlet distribution of the matrix sensing task is shown in Figure 3. The smallest eigenvalue at the converged point for each algorithm is shown in Table 2 and Table 3.
(b)  $d = 50$  , toroidal