SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \boldsymbol {a} _ {t} ^ {\top} \boldsymbol {a} ^ {*} = \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} \boldsymbol {a} _ {0} ^ {\top} \boldsymbol {a} ^ {*} + \frac {1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t}}{k} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} A _ {0} \\ + \frac {\eta \| \boldsymbol {a} ^ {*} \| ^ {2}}{2 \pi} \sum_ {s = 0} ^ {t - 1} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - 1 - s} \frac {1}{N} \sum_ {i = 1} ^ {N} B _ {s} ^ {i} + S \left(\epsilon_ {0: t - 1}\right), \tag {5} \\ \end{array}
$$
where  $S(\epsilon_{0:t-1})$  (defined in Appendix A) is the noise term involving  $\epsilon_0, \dots, \epsilon_{t-1}$  and
$$
B _ {s} ^ {i} = g (\phi_ {s} ^ {i}) - 1 + \frac {\eta}{2 \pi} \frac {| \mathbf {1} ^ {\top} \mathbf {a} ^ {*} | ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \sum_ {l = 0} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} (\pi - g (\phi_ {l} ^ {i})).
$$
α	1/64	1/32	1/16	1/8	1/4	1/2
Local SGD	0.61	0.62	0.69	0.93	1.00	1.00
Minibatch SGD	0.60	0.62	0.69	0.91	1.00	1.00
Table 1: Success probabilities of converging to the global minima under different values of  $\alpha = |\mathbf{1}^\top \mathbf{a}^*|^2 / (k\|\mathbf{a}^*\|^2)$ . The simulation setting is given in Section 5.
Notice that the first term  $g(\phi_s^i) - 1$  in  $B_{s}^{i}$  is negative whenever  $\phi_s^i < \pi /2$ , but the second term in  $B_{s}^{i}$  is always positive for  $\phi_s^i\in (0,\pi ]$ . In particular,  $\pi -g(\phi_l^i)$  tends to be larger when  $\phi_l^i$  gets closer to  $\pi$  (i.e.,  $w_{t}^{i}$  drifts away from  $\pmb{w}^{*}$ ). This insight provides a possibility that local SGD can correct the signal of the second layer by itself, instead of injecting additional noise like [63].
Through carefully inspecting the ingredients of dynamics, we have the following roadmap to show the global convergence of local SGD: (1) For arbitrary initialization  $(\pmb{v}_0, \pmb{a}_0)$  except for a measure zero set, where the angle of the first layer between initialization and the global minimum is  $\pi$ , show that  $\pmb{a}_t^\top \pmb{a}^*$  can turn to the positive signal in polynomial time; (2) Show that  $\pmb{a}_t^\top \pmb{a}^*$  can be lower bounded by a positive constant value and  $\phi_t$  will decrease below  $\pi/2$ . (3) After entering the attraction basin, show that  $(\pmb{w}_t, \pmb{a}_t)$  will converge to the ground truth with a linear speedup guarantee.
4.2 Self-correction of Signals in Two Layers
In this subsection, we will show the iterates of local SGD can enter the attraction basin such that  $\pmb{a}_t^\top \pmb{a}^* > 0$  and  $\phi_t < \pi/2$  after  $\tilde{O}(\eta^{-1})$  steps. Before that, we introduce an assumption on  $\pmb{a}^*$ .
Assumption 1. Define  $\alpha = |\mathbf{1}^{\top}\mathbf{a}^{*}|^{2} / (k\| \mathbf{a}^{*}\|^{2})$ . We assume  $k\geq 320(\pi -1)^{2}$  and the ground truth satisfies  $|\mathbf{1}^{\top}\mathbf{a}^{*}|^{2}\leq \frac{k(\pi + k - 1)}{720\pi\log(4 + k^{2})}$  and
$$
\alpha > \frac {\pi + k - 1}{(\pi - 1) k} \left(1 - \frac {3 2 (\pi - 1)}{k}\right) ^ {- 1}. \tag {6}
$$
The conditions on  $k$  and  $|\mathbf{1}^{\top}\mathbf{a}^{*}|^{2}$  are imposed for technical reasons. We believe that a constant lower bound (the right-hand side of (6)) for  $\alpha$  is necessary to show the global convergence with almost arbitrary initialization, which is also verified by our simulation results in Table 1. We compute convergence probabilities of local SGD and minibatch SGD under different values of  $\alpha$ . The initial points are randomly selected by  $v_{0} \sim \mathrm{Unif}(\mathbb{S}^{d - 1})$  and  $a_0 \sim \mathrm{Unif}(\mathbb{B}^k(|\mathbf{1}^\top \mathbf{a}^*| / \sqrt{k}))$  in each trial. If the convergence probability reaches 1, it means that local SGD or minibatch SGD can converge to the global minima with arbitrary initialization except for a measure zero set. When  $\alpha \leq 1/8$ , both minibatch SGD and local SGD cannot converge to the global minima with probability 1.
Theorem 1 (Self-correction of the second layer). For any initial point  $(\pmb{v}_0, \pmb{a}_0)$  with  $\phi_0 \in [0, \pi)$ , we denote  $\tau_a = \inf \{t \geq 0 : \pmb{a}_t^\top \pmb{a}^* \geq \gamma_a\}$  the first time, where  $\gamma_a = \frac{16(\pi - 1)|\mathbf{1}^\top \pmb{a}^*|^2}{k(\pi + k - 1)}$ . Under Assumption 1, if
$$
c k ^ {2} \sqrt {\log (N d k / \delta)} \max  \left\{\eta (\sqrt {I k} + I), \eta (\sqrt {I d} + I), \sqrt {\frac {\eta k}{N}} \right\} \frac {\| \boldsymbol {a} ^ {*} \| ^ {2}}{\left| \mathbf {1} ^ {\top} \boldsymbol {a} ^ {*} \right| ^ {2}} <   1, \tag {7}
$$
for a sufficiently large constant  $c$ , then  $\tau_{a} \leq O(\eta^{-1} \log k)$  with probability at least  $1 - \delta$ .
This theorem completes the first step of the roadmap, the self-correction of the signal in the second layer, whose proof is given in Appendix D.1. If  $|\mathbf{1}^{\top}\mathbf{a}^{*}|^{2} / \| \mathbf{a}^{*}\|^{2}\leq 1 / \mathrm{poly}(p)$ , Du et al. [13] proved GD can converge to the spurious local minima with the initial condition  $a_0^\top a^* < 0$  and  $\phi_0 > \pi /2$ . Therefore, our results do not contradict theirs because we assume  $|\mathbf{1}^{\top}\mathbf{a}^{*}|^{2} / \| \mathbf{a}^{*}\|^{2}$  is large. The simulations of [13, 63] show that the success probability of converging to the global optimum of SGD increases as the ratio  $|\mathbf{1}^{\top}\mathbf{a}^{*}| / \| \mathbf{a}^{*}\|$  increasing. Theorem 1 can potentially explain this phenomenon since the condition (6) will be satisfied eventually when  $|\mathbf{1}^{\top}\mathbf{a}^{*}| / \| \mathbf{a}^{*}\|$  keeps increasing.
Proof sketch of Theorem 1. The proof is very technical since the angle  $\phi_t^i$  will affect the sign of  $B_s^i$  in the dynamic (5) of  $\pmb{a}_t^\top \pmb{a}^*$ , while controlling  $\phi_t^i$  also requires bounding the scale and controlling
the sign of  $\pmb{a}_t^\top \pmb{a}^*$ . To tackle this challenge: (1) We divide the initial region of  $\phi_0$  into four regions:  $(0, \tilde{\phi}^o), [\tilde{\phi}^o, \pi/2), [\pi/2, \tilde{\phi}^u)$  and  $[\tilde{\phi}^u, \pi)$ ; (2) For each of the first three initial regions, we prove that either  $\pmb{a}_t^\top \pmb{a}^* \geq \gamma_a$  in  $\widetilde{O}(1/\eta)$  steps or enter the next region in  $\widetilde{O}(1/\eta)$  steps; (3) In the last region  $[\tilde{\phi}^u, \pi)$ , we use the condition (6) to show  $\pmb{a}_t^\top \pmb{a}^*$  will turn positive in  $\widetilde{O}(1/\eta)$  steps. The dividing technique enables us to control the individual angle  $\phi_t^i$  in four "nice" regions where  $\pmb{a}_t^\top \pmb{a}^*$  can be corrected with negligible disturbance. Here  $\tilde{\phi}^o = \arccos(1/5)$  is chosen for convenience to avoid dependence on the initial value  $\sin \phi_0$  if  $\phi_0 < \pi/2$ , and  $\tilde{\phi}^u$  is carefully chosen for technical reasons.
Lemma 3. Denote  $\tilde{\phi}^u = \arccos \left(-\frac{144\pi}{\pi + k - 1}\frac{|\mathbf{1}^\top\mathbf{a}^*|^2}{k}\log (4 + k^2)\right)$  and
$$
\varpi = \min \left\{\sin \tilde {\phi} ^ {u} \mathbb {1} _ {\phi_ {0} \leq \tilde {\phi} ^ {u}} + \sin \phi_ {0} \mathbb {1} _ {\phi_ {0} > \tilde {\phi} ^ {u}}, \sin \left[ \pi - \left\{\left(2 0 \| \boldsymbol {a} ^ {*} \| ^ {2}\right) ^ {- 1} \wedge 1 \right\} \right] \right\},
$$
Under the same conditions of Theorem 1. If the learning rate satisfies
$$
c \| \boldsymbol {a} ^ {*} \| ^ {2} \sqrt {\log (N d k / \delta)} \max  \left\{\eta (\sqrt {I d} + I), \sqrt {\frac {\eta}{N}} \right\} \leq \frac {\varpi^ {2}}{1 6 k}, \tag {8}
$$
for a large absolute constant  $c > 0$ , we have  $\sin \phi_{\tau_a} \geq \frac{\varpi}{15\sqrt{k}}$  with probability at least  $1 - \delta$ .
Recalling the dynamic (4), we know  $\sin \phi_t$  tends to decrease to 0 when  $\phi_t > \pi /2$  and  $a_{t}^{\top}a^{*} < 0$ . It means that the first layer moves in the direction of spurious local minima during the self-correction process of the second layer. Lemma 3 provides a lower bound for  $\sin \phi_{\tau_a}$  to guarantee the first layer will not converge to the local minima. The proof of Lemma 3 is deferred to Appendix D.2.
Theorem 2 (Self-correction of the first layer). For the initial point  $(\pmb{v}_0, \pmb{a}_0)$  with  $\pmb{a}_0^\top \pmb{a}^* \geq \gamma_a$  (defined in Theorem 1) and  $\sin \phi_0 > 0$ , we denote  $\tau_v = \inf \{t \geq 0 : \phi_v \leq \tilde{\phi}^l\}$ , where  $\cos \tilde{\phi}^l = \frac{144\pi}{\pi + k - 1} \frac{|\pmb{1}^\top \pmb{a}^*|^2}{k} \log (4 + k^2)$ . If the learning rate satisfies
$$
c \left\| \boldsymbol {a} ^ {*} \right\| ^ {2} \sqrt {\log (N d k / \delta)} \max  \left\{\eta (\sqrt {I d} + I), \sqrt {\frac {\eta}{N}} \right\} \leq \sin^ {2} \phi_ {0}, \tag {9}
$$
for a large absolute constant  $c > 0$ , it holds that  $\tau_v \leq O\left(\frac{1}{\eta}\frac{k^2}{\sin^3\phi_0}\right)$  and  $\pmb{a}_{\tau_v}^\top \pmb{a}^* \geq \gamma_a / 32$  with probability at least  $1 - \delta$ .
This theorem finishes the second step of the roadmap to convergence. The detailed proof is deferred to Appendix D.3. Now we give the iteration complexity spent for self-correction for the worst initialization:  $\pmb{a}_0^\top \pmb{a}^* < 0$  and  $\phi_0 > \max \{\tilde{\phi}^u, \pi - (\| \pmb{a}^* \|^{-2} / 20 \wedge 1)\}$ . Since  $\sin \tilde{\phi}^u = O(1)$ , plugging the lower bound of  $\sin \phi_{\tau_a}$  in Lemma 3, we can get
$$
\tau_ {a} + \tau_ {v} \leq O \left(\frac {1}{\eta} \frac {k ^ {7 / 2}}{\sin^ {3} \phi_ {0}}\right). \tag {10}
$$
Proof sketch of Theorem 2. We divide the initial region into three regions:  $(\tilde{\phi}^f, \pi)$ ,  $[\tilde{\phi}^f, \tilde{\phi}^l)$  and  $[0, \tilde{\phi}^l]$ , where  $\tilde{\phi}^f$  is adapted to the choices for  $\tilde{\phi}^l$  and  $\tilde{\phi}^u$ . For the first two regions, we show that  $\phi_t$  will enter the next region in polynomial time, while the last region is our target. The difficult part to show the correction of  $\phi_t$  falls in how to guarantee  $a_t^\top a^*$  will be lower bounded by  $O(\gamma_a)$  for any  $t \geq 0$ , which is divided and conquered by considering the intermediate quantities such as  $\tilde{\phi}^l$ ,  $\tilde{\phi}^f$  and  $\tilde{\phi}^u$ . To better understand the choices for these quantities and  $\gamma_a$ , we rearrange  $B_s^i$  and have the following decomposition
$$
\begin{array}{l} B _ {s} ^ {i} = \underbrace {\frac {\pi - 1}{\pi + k - 1} \frac {\left| \mathbf {1} ^ {\top} \boldsymbol {a} ^ {*} \right| ^ {2}}{\left\| \boldsymbol {a} ^ {*} \right\| ^ {2}} - 1} _ {\zeta_ {a}} - \underbrace {\frac {\pi - 1}{\pi + k - 1} \frac {\left| \mathbf {1} ^ {\top} \boldsymbol {a} ^ {*} \right| ^ {2}}{\left\| \boldsymbol {a} ^ {*} \right\| ^ {2}} \left(1 - \frac {\eta (\pi + k - 1)}{2 \pi}\right) ^ {s}} _ {\varrho_ {a}} \\ + \underbrace {g (\boldsymbol {\phi} _ {s} ^ {i}) - \frac {\eta}{2 \pi} \frac {| \mathbf {1} ^ {\top} \boldsymbol {a} ^ {*} | ^ {2}}{\| \boldsymbol {a} ^ {*} \| ^ {2}} \sum_ {l = 0} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} (1 - g (\boldsymbol {\phi} _ {l} ^ {i}))} _ {\vartheta_ {a}}. \\ \end{array}
$$
Due to the condition (6) in Assumption 1,  $\zeta_{a}$  is positive and used to correct the sign of  $\pmb{a}_{t}^{\top}\pmb{a}^{*}$  and keep it positive. Since  $\varrho_{a}$  is still not negligible when  $s$  is small, we use  $\gamma_{a}$  to compensate  $-\varrho_{a}$  after enrolled summation. The uncertain part  $\vartheta_{a}$  will be positive whenever  $\tilde{\phi}^l\leq \phi_l^i\leq \tilde{\phi}^u$  for  $l\leq s$
4.3 Convergence with Linear Speedup
With the correction guarantees in the previous subsection, we are ready to proceed with the convergence analysis of local SGD.
Theorem 3. Suppose the initial point  $(\pmb{v}_0, \pmb{a}_0)$  satisfies  $\pmb{a}_0^\top \pmb{a}^* \geq \gamma_a / 32$  and  $\phi_0 \leq \tilde{\phi}^l$ . For any  $\epsilon > 0$ , we choose  $\eta = \frac{1}{ck^2 \| \pmb{a}^* \|^2} \frac{N\epsilon}{d\log(dN / \epsilon\delta)}$  for some absolute constant  $c > 0$ . If  $I \lesssim \frac{d}{\epsilon N} \min \left\{1, \frac{k^2 d^{1/2}}{N^{1/2}}, \frac{k^4 d}{N^2\epsilon}\right\}$  and  $\epsilon < \min \{d^{-1}, dk^{-2}\}$ , then  $\ell(\pmb{v}_T, \pmb{a}_T) \lesssim \epsilon$  holds with probability at least  $1 - \delta$  where  $T = \widetilde{O}\left(\frac{dk^4}{N\epsilon}\right)$ .
We have the following implications about the result in Theorem 3:
(1) To the best of our knowledge, this is the first convergence result with linear speedup on the number of machines  $N$  for two-layer neural networks. Besides, our convergence analysis does not rely on the overparameterization of the width of the second layer (i.e.,  $k$ ).
(2) The dependency on  $\epsilon$  matches the best-known results of local SGD for strongly convex objective in Woodworth et al. [53]. In fact, the size of the first layer  $d$  resembles the variance of stochastic gradients  $\sigma^2$  in the traditional optimization literature. We can show  $\langle \nabla_{\boldsymbol{a}}L(\boldsymbol{w}_t,\boldsymbol{a}_t),\boldsymbol{a}_t - \boldsymbol{a}^*\rangle \geq \frac{\pi - 1}{2\pi}\| \boldsymbol{a}_t - \boldsymbol{a}^*\|^2 - O(\phi_t)\| \boldsymbol{a}^*\|^2$ . Therefore, when the first layer converges has converged ( $\phi_t\approx 0$ ),  $\boldsymbol{a}_t$  can converge to  $\boldsymbol{a}^*$  like the strongly convex regime.
(3) According to the condition on  $I$ , we can obtain the communication complexity in the convergence stage as
$$
R _ {\text {c o n v}} = \frac {T}{I} = \widetilde {O} \left(\max  \left\{k ^ {4}, k ^ {2} \sqrt {\frac {N}{d}}, \frac {N ^ {2} \epsilon}{d} \right\}\right). \tag {11}
$$
When  $N \lesssim \min\{dk^4, \epsilon^{-1}\}$ , the communication complexity in this stage can be  $R_{\mathrm{conv}} = \widetilde{O}(k^4)$ , which is significantly reduced compared with the iteration complexity.
We show local SGD's convergence layer by layer. Next lemma ensures that the weight of the first layer can converge to the ground truth in polynomial time.
Lemma 4 (Convergence of the first layer). Under the settings in Theorem 3. Suppose the initial point satisfies  $\phi_0 \leq \tilde{\phi}^l$  and  $\mathbf{a}_0^\top \mathbf{a}^* \geq \gamma_a / 32$ . With probability at least  $1 - \delta$ , we can guarantee that  $\sin^2 \phi_t \lesssim \epsilon$  holds for any  $\widetilde{O}\left(k^2 \eta^{-1}\right) \leq t \leq \widetilde{O}(\eta^{-2})$ .
It is worthwhile noticing that  $(\pmb{v}_{\tau_v},\pmb{a}_{\tau_v})$  in Theorem 2 satisfies the conditions for initial point in Lemma 4. Therefore, local SGD enters the attraction basin after finishing the self-correction process. In fact, showing the complexity  $\widetilde{O} (\epsilon^{-1})$  is not trivial based on the traditional analysis of local SGD [47, 54]. The issue comes from the inner-product noise term  $M_{1,t}$  and the discrepancy term  $H_{t}$  in Lemma 1, whose enrolled summations can only be bounded by  $O(\sqrt{\epsilon / d})$  and  $O\{(\sqrt{Id} +I)\epsilon \}$  respectively at the beginning of convergence stage. Thanks to the special structure of gradient, we find that the scales of  $M_{1,t}$  and  $H_{t}$  can shrink as  $\sin \phi_t$  decreases. In light of this, we can continuously refine the bound by the following contraction: for  $(K + 1)T_{v}\leq t\leq \widetilde{O} (\eta^{-2})$  it holds that
$$
\sin^ {2} \phi_ {t} \lesssim \max  \left\{\left(\sqrt {\frac {\epsilon}{d}} + (\sqrt {I d} + I) \epsilon\right) ^ {1 + \frac {K}{2}}, \epsilon \right\}.
$$
By taking  $K = O(\log (1 / \epsilon))$ , we can obtain the target convergence  $\sin^2\phi_t\lesssim \epsilon$  with high probability. More details can be found in Appendix E.1.
Lemma 5 (Convergence of the second layer). Under the choice for  $\eta$  and conditions for  $\epsilon$  in Theorem 3. Suppose  $\sin^2\phi_t\leq \epsilon$  holds for any  $0\leq t\leq \widetilde{O} (\eta^{-2})$ . With probability at least  $1 - \delta$ , we can guarantee that  $\| \pmb {a}_t - \pmb {a}^*\| ^2\lesssim \epsilon$  holds for any  $\widetilde{O}\left(\eta^{-1}\right)\leq t\leq \widetilde{O} (\eta^{-2})$ .
The lemma stated above guarantees the convergence of the second layer after that of the first layer, whose proof is deferred to Appendix E.2. Equipped with Lemmas 4 and 5, we can prove Theorem 3 by leveraging the closed form of the objective  $L(\boldsymbol{w}, \boldsymbol{a})$  (see Lemma 6).
(a)  $\pmb{a}_0 = -\pmb{a}^*, \phi_0 > \pi / 2$
(b)  $\pmb{a}_0 = -\pmb{a}^*, \phi_0 < \pi / 2$
(c)  $\pmb{a}_0 = \mathbf{0}, \phi_0 > \pi / 2$
Figure 1: Converged trajectories of local SGD and minibatch SGD with different bad initial points. The dimensions of the two layers are  $k = 10$  and  $d = 25$ . The number of skipped communication is  $I = 8$ . The batch size is 4. The two algorithms' learning rates are  $\eta = 0.1$ .
The perturbed GD method in Zhou et al. [63] requires manually adjusting the scale of injected noises and the learning rate between the transition of two phases. In local SGD, we can use a universal learning rate in the self-correction stage and convergence stage. Considering an initial point that is closer to spurious local minima  $(\phi_0 > \tilde{\phi}^u)$  defined in Lemma 3, conditions (7), (8) and (9) on the learning rate can be satisfied if we choose  $I = O\{d / (\epsilon N)\}$ . Together with (10) and (11), if  $N \lesssim \min\{dk^4, \epsilon^{-1}\}$ , we can get the total communication complexity
$$
R _ {\text {t o t a l}} = \widetilde {O} \left(\frac {k ^ {1 1 / 2}}{\sin^ {3} \phi_ {0}}\right).
$$
Therefore, our theory shows that local SGD can correct the signals and converge to the global minima with almost arbitrary initialization except for the case of initializing at the local minima  $(\phi_0 = \pi)$ .
We now report some simulation results on synthetic data. We also compare the performance of two algorithms: local SGD and minibatch SGD. At each round, minibatch SGD updates the model weights by using the stochastic gradients with batch size  $N I$  in total to update the model, where each local machine computes  $I$  gradients and communicates with other machines. minibatch SGD and local SGD have the same computation and communication structure [54].
In our first simulation, we set  $\| \pmb{w}^{*} \| = \| \pmb{a}^{*} \| = 1$  and  $|1^{\top} \pmb{a}^{*}| = \sqrt{k}$ . The results starting from three bad initial regions (i.e.,  $\phi_0 > \pi / 2$  or  $\pmb{a}_0^\top \pmb{a}^* \leq 0$ ) are reported in Figure 1. As we can see, the signals of two layers can be both corrected to a good region (i.e., both  $\cos(\phi_t)$  and  $\pmb{a}_t^\top \pmb{a}^*$  go to 1 and the loss goes to 0 when  $t$  increases) even in the worst case when  $\pmb{a}_0 = -\pmb{a}^*$  and  $\phi_0 > \pi / 2$ . An interesting phenomenon is that local SGD can correct the signals faster than minibatch SGD. The reason is that the statistical error of stochastic gradients is not dominating in the self-correction process, so the effect of large batch size in minibatch SGD (i.e., with batch size  $N I$  for 1 iteration) is not as useful as smaller batch size in local SGD with more iterations (i.e., batch size 1 for  $I$  iterations on each machine with  $N$  machines in total). These results corroborate our theoretical analysis.
(a)  $\alpha = 1 / 32$
(b)  $\alpha = 1 / 16$
(c)  $\alpha = 1 / 4$
Figure 2: Trajectories of local SGD and minibatch SGD under different values of  $\alpha$ . The dimensions of the two layers are  $k = 64$  and  $d = 25$ . The number of skipped communication is  $I = 8$ . The batch size is 16. The two algorithms' learning rates are  $\eta = 0.1$ .
In the second simulation, we plot the trajectories of local SGD and minibatch SGD under different values of  $\alpha$  in Figure 2. Here we set  $\pmb{w}^{*} = \mathbf{1}_{d} / \sqrt{d}$  and  $\pmb{a}^{*} = (\mathbf{1}_{\alpha k},0,\dots ,0) / \sqrt{\alpha k}$ . The bad initial point is fixed as  $\pmb{v}_0 = (-1,0,\dots ,0)$  and  $\pmb{a}_0 = (-\sqrt{\alpha},0,\dots ,0)$ , where  $\phi_0 > \pi /2$  and  $\pmb{a}_0^\top \pmb{a}^* < 0$ . When  $\alpha = 1/32,1/16$ , we can see that minibatch SGD and local SGD converge to the local minima. When  $\alpha = 1/4$ , they can converge to the global minima with the same initial point. In this case, as we can see from Figure 2(c), the signals of two layers can be both corrected to a good region.
In the third simulation, we calculate the probabilities that local SGD and minibatch SGD converge to the global minimum under different values of  $\alpha$ . The averaged results are taken over 100 independent repeated simulations. In each trial, we generate initial points by  $v_0 \sim \mathrm{Unif}(\mathbb{S}^{d-1})$  and  $a_0 \sim \mathrm{Unif}(\mathbb{B}^k(|\mathbf{1}^\top \mathbf{a}^*|/\sqrt{k}))$ . The results are given in Table 1.
We theoretically investigate the convergence of local SGD, a cornerstone algorithm in federated learning with strong empirical performance. We demonstrate convergence to the global minimum for two-layer neural networks without overparameterization, without injecting noise, and when the input data is Gaussian. A new self-correction mechanism guarantees the algorithm reaches a good region even if the initialization is in a bad region. The landscape of the objective is divided into several regions to carefully control the interference of the two layers during the correction process. A new exact recursive characterization of the direction of global parameter provides the key to show convergence to the global minimum with linear speedup in the number of machines and reduced communication rounds. Experiments on simulated data corroborate the theoretical results. To the best of our knowledge, this work is the first to theoretically demonstrate the global convergence of the vanilla local SGD for neural networks without overparameterization.
We would like to thank the anonymous reviewers for their helpful comments. Mingrui Liu is supported by a grant from George Mason University. The work of Yajie Bao was done when he was virtually visiting Mingrui Liu's research group in the Department of Computer Science at George Mason University.