SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \epsilon_ {t, i} - \epsilon_ {t, i} ^ {\prime} \right\| \leq 2 L \left\| w _ {t + 1} - w _ {t} \right\| + \left(2 \beta L + 3 \rho C _ {d}\right) \left\| w _ {t} \right\| + 3 \rho C _ {d} \left\| w _ {t + 1} \right\| \tag {45}
$$
Applying Azuma-Hoeffding inequality to Eq. (42), with Eq. (45) we can obtain
$$
\begin{array}{l} \| \nu_ {t} \| ^ {2} \leq \frac {4 \log (4 / \delta)}{n b _ {1}} \sum_ {\tau = s} ^ {t - 1} [ 2 L \| w _ {\tau + 1} - w _ {\tau} \| + (2 \beta L + 3 \rho C _ {d}) \| w _ {\tau} \| + 3 \rho C _ {d} \| w _ {\tau + 1} \| ] ^ {2} \\ \leq \frac {4 8 \log (4 / \delta)}{n b _ {1}} \sum_ {\tau = s + 1} ^ {t} \left(L ^ {2} \| w _ {\tau} - w _ {\tau - 1} \| ^ {2} + 5 \rho^ {2} C _ {d} ^ {2} \| w _ {\tau} \| ^ {2}\right) \tag {46} \\ \end{array}
$$
since  $\beta$  is  $\Theta (\epsilon^{1 + \theta})$  and  $C_d$  is  $\Theta (\epsilon^{1 - \alpha})$ . According to Eq. (34), we have
$$
\begin{array}{l} L \left\| w _ {\tau} - w _ {\tau - 1} \right\| \\ = L \| - \eta \mathcal {H} (I - \eta \mathcal {H}) ^ {\tau - s - 2} w _ {s + 1} - \eta \sum_ {\tau^ {\prime} = s + 1} ^ {\tau - 2} \eta \mathcal {H} (I - \eta \mathcal {H}) ^ {\tau^ {\prime} - s - 2} \left(\bar {\Delta} _ {\tau^ {\prime}} w _ {\tau^ {\prime}} + \nu_ {\tau^ {\prime}} + \zeta_ {\tau^ {\prime}}\right) \\ + \eta (\bar {\Delta} _ {\tau - 1} w _ {\tau - 1} + \nu_ {\tau - 1} + \zeta_ {\tau - 1}) \| \\ \leq L \eta \gamma (1 + \eta \gamma) ^ {\tau - s - 2} p _ {s} r _ {0} + \frac {L \eta \gamma}{2} (1 + \eta \gamma) ^ {\tau - s - 2} p _ {s} r _ {0} + L \eta \| \bar {\Delta} _ {\tau - 1} w _ {\tau - 1} + \nu_ {\tau - 1} + \zeta_ {\tau - 1} \| \\ \leq 2 L \eta \gamma (1 + \eta \gamma) ^ {\tau - s - 2} p _ {s} r _ {0} + L \eta \| \bar {\Delta} _ {\tau - 1} w _ {\tau - 1} + \nu_ {\tau - 1} + \zeta_ {\tau - 1} \| \tag {47} \\ \end{array}
$$
In the first inequality, the first term is derived by the definition of  $w_{s+1}$ . The second term is derived by the supposition that Eq. (38) holds for  $t \leq t_0$  and the fact that Eq. (38) implies
$$
\eta \sum_ {\tau = s + 1} ^ {t - 1} (1 + \eta \gamma) ^ {t - \tau - 1} \| \bar {\Delta} _ {\tau} w _ {\tau} + \nu_ {\tau} + \zeta_ {\tau} \| \leq \frac {1}{2} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \tag {48}
$$
Combining Eq. (46) and Eq. (47), we have
$$
\begin{array}{l} \| \nu_ {t} \| ^ {2} \leq \frac {4 8 \log (4 / \delta)}{n b _ {1}} \sum_ {\tau = s + 1} ^ {t} (L ^ {2} \| w _ {\tau} - w _ {\tau - 1} \| ^ {2} + 5 \rho^ {2} C _ {d} ^ {2} \| w _ {\tau} \| ^ {2}) \\ \leq \frac {2 7 0 \log (4 / \delta) \rho^ {2} C _ {d} ^ {2}}{n b _ {1} \eta \gamma} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} + \frac {1 9 2 \log (4 / \delta) L ^ {2} \eta \gamma}{n b _ {1}} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} \\ + \frac {9 6 \log (4 / \delta) L ^ {2} \eta^ {2}}{n b _ {1}} \sum_ {\tau = s + 1} ^ {t - 2} \| \bar {\Delta} _ {\tau} w _ {\tau} + \nu_ {\tau} + \zeta_ {\tau} \| ^ {2} \\ \leq \frac {3 0 0 \log (4 / \delta) \rho^ {2} C _ {d} ^ {2}}{n b _ {1} \eta \gamma} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} + \frac {1 9 2 \log (4 / \delta) L ^ {2} \eta \gamma}{n b _ {1}} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} \\ + \frac {4 \log (4 / \delta) L ^ {2}}{n b _ {1} \eta \gamma C _ {T} ^ {2}} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} + \frac {5 0 0 0 \log^ {2} (4 / \delta) L ^ {4} \eta^ {2} p _ {s} ^ {2} r _ {0} ^ {2}}{b _ {1} ^ {2}} \sum_ {\tau = s + 1} ^ {t - 2} \frac {(1 + \eta \gamma) ^ {2 (\tau - s - 1)}}{(\tau - s) ^ {2}} \\ \leq \frac {1}{2 8 8 \eta^ {2} C _ {T} ^ {2}} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} + \frac {8 0 0 \log^ {2} (4 / \delta) L ^ {2} \eta \gamma}{n b _ {1}} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} \\ + \frac {5 0 0 0 \log^ {2} (4 / \delta) L ^ {4} \eta^ {2} p _ {s} ^ {2} r _ {0} ^ {2}}{b _ {1} ^ {2}} \sum_ {\tau = s + 1} ^ {s + \frac {L \eta}{b _ {1} \gamma}} \frac {(1 + \eta \gamma) ^ {2 (\tau - s - 1)}}{(\tau - s) ^ {2}} \\ \leq \frac {1 0 0 0 0 \log (4 / \delta) L ^ {4} \eta^ {4}}{b _ {1} ^ {4} \gamma^ {2}} \cdot \frac {(1 + \eta \gamma) ^ {2 (t - s - 1)} L ^ {2} p _ {s} ^ {2} r _ {0} ^ {2}}{(t - s) ^ {2}} + \frac {1}{1 4 4 \eta^ {2} C _ {T} ^ {2}} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} \\ \leq \frac {(1 + \eta \gamma) ^ {2 (t - s - 1)} L ^ {2} p _ {s} ^ {2} r _ {0} ^ {2}}{(t - s) ^ {2}} + \frac {1}{1 4 4 \eta^ {2} C _ {T} ^ {2}} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} \tag {49} \\ \end{array}
$$
The exponential term in Eq. (40) can be addressed by the following strategy. When  $t \geq \tilde{O}\left(\frac{1}{1 - \lambda}\right)$ , the term can be dominated by other terms such as  $\frac{1}{\eta C_T}$ . When  $t < \tilde{O}\left(\frac{1}{1 - \lambda}\right)$ , it can be bounded by
$$
\frac {L ^ {2} \eta^ {2} \log (4 / \delta) p _ {s} r _ {0} ^ {2}}{n (1 - \lambda) b _ {1}} \leq \frac {L ^ {2} \eta^ {2} (t - s) ^ {2} \log (4 / \delta) p _ {s} ^ {2} r _ {0} ^ {2}}{(1 - \lambda) b _ {1} (t - s) ^ {2}} \tag {50}
$$
The term  $t - s$  in the numerator will be bounded by  $\eta$  in this case and hence it can be merged to the first term in Eq. (39). In the third inequality of Eq. (49), we split the last term into two parts:  $\tau - s > \frac{L\eta}{b_1\gamma}$  and  $\tau - s \leq \frac{L\eta}{b_1\gamma}$ . Since  $\int_t^{+\infty} \frac{dx}{x^2} = \frac{1}{t}$ , we can merge the case  $\tau - s > \frac{L\eta}{b_1\gamma}$  into the second term and estimate the rest one where  $\tau - s$  is small. According to the choice of  $\theta$ , we have  $b_1 \geq \Theta (\epsilon^{2 - \theta - 5\alpha})$  and  $\frac{\eta^2 C_d^2 C_T^3}{b_1} \leq O(1)$  and hence get the estimation in Eq. (49). We should notice that we use the relation  $\frac{\eta}{b_1\gamma} \leq O(1)$  in our proof, which is automatically satisfied. By Eq. (49) we can reach the conclusion in Eq. (39). Furthermore, we have
$$
\begin{array}{l} \eta \sum_ {\tau = s + 1} ^ {t - 1} (1 + \eta \gamma) ^ {t - \tau - 1} \| \nu_ {\tau} \| \\ \leq L \eta (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} (\sum_ {\tau = s + 1} ^ {t - 1} \frac {1}{\tau - s}) + \frac {1}{1 2} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \\ \leq L \eta \log \left(C _ {T}\right) (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} + \frac {1}{1 2} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \leq \frac {1}{6} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \tag {51} \\ \end{array}
$$
The last step to prove Eq. (38) is to estimate the term corresponding to  $\zeta_t$ , which is a new term only occurred in decentralized algorithms. Recall the definitions in Eq. (43), we have
$$
\begin{array}{l} \zeta_ {t} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \left(\mathcal {H} _ {t} ^ {(i)} + \hat {\Delta} _ {t, 1} ^ {(i)}\right) w _ {t} ^ {(i)} - \left(\mathcal {H} _ {t} ^ {(i)} + \tilde {\Delta} _ {t, 1} ^ {(i)}\right) w _ {t} - (1 - \beta) \left(\left(\mathcal {H} _ {t} ^ {(i)} + \hat {\Delta} _ {t, 2} ^ {(i)}\right) w _ {t - 1} ^ {(i)} - \left(\mathcal {H} _ {t} ^ {(i)} + \tilde {\Delta} _ {t, 2} ^ {(i)}\right) w _ {t - 1}\right) \right] \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {H} _ {t} ^ {(i)} [ (w _ {t} ^ {(i)} - w _ {t}) - (1 - \beta) (w _ {t - 1} ^ {(i)} - w _ {t - 1}) ] + \frac {1}{n} \sum_ {i = 1} ^ {n} \hat {\Delta} _ {t, 1} ^ {(i)} (w _ {t} ^ {(i)} - w _ {t}) \\ + \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\hat {\Delta} _ {t, 1} ^ {(i)} - \tilde {\Delta} _ {t, 1} ^ {(i)}\right) w _ {t} - \frac {1 - \beta}{n} \sum_ {i = 1} ^ {n} \hat {\Delta} _ {t, 2} ^ {(i)} \left(w _ {t - 1} ^ {(i)} - w _ {t - 1}\right) - \frac {1 - \beta}{n} \sum_ {i = 1} ^ {n} \left(\hat {\Delta} _ {t, 2} ^ {(i)} - \tilde {\Delta} _ {t, 2} ^ {(i)}\right) w _ {t - 1} \tag {52} \\ \end{array}
$$
Then by Assumption 3, Assumption 4, Eq. (37), Lemma 10 and Cauchy-Schwartz inequality, we have
$$
\begin{array}{l} \| \zeta_ {t} \| ^ {2} \leq \frac {4 L ^ {2}}{n} (\| X _ {t} - \bar {X} _ {t} - (X _ {t} ^ {\prime} - \bar {X} _ {t} ^ {\prime}) \| _ {F} ^ {2} + \| X _ {t - 1} - \bar {X} _ {t - 1} - (X _ {t - 1} ^ {\prime} - \bar {X} _ {t - 1} ^ {\prime}) \| _ {F} ^ {2}) \\ + 1 4 4 \rho^ {2} C _ {d} ^ {2} \left(\| w _ {t} \| ^ {2} + \| w _ {t - 1} \| ^ {2}\right) \tag {53} \\ \end{array}
$$
It is sufficient to prove
$$
\begin{array}{l} \frac {L}{\sqrt {n}} \| X _ {t} - \bar {X} _ {t} - (X _ {t} ^ {\prime} - \bar {X} _ {t} ^ {\prime}) \| _ {F} \leq 2 (\frac {1 + \lambda^ {2}}{2}) ^ {\frac {t - s - 1}{2}} L \sqrt {p _ {s}} r _ {0} + \frac {1}{4 8 \eta C _ {T}} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \\ + \frac {L \eta (1 + \eta \gamma) ^ {t - s - 1} L p _ {s} r _ {0}}{4 \sqrt {b _ {1}} (t - s)} \tag {54} \\ \end{array}
$$
because of Eq. (53) and the parameter setting. Eq. (54) can also be proven by induction. When  $t = s + 1$  the condition is satisfied. Next we will estimate  $\| X_{t} - \bar{X}_{t} - (X_{t}^{\prime} - \bar{X}_{t}^{\prime})\|_{F}^{2}$ . By Assumption 5 and Young's inequality we have
$$
\begin{array}{l} \left\| X _ {t} - \bar {X} _ {t} - \left(X _ {t} ^ {\prime} - \bar {X} _ {t} ^ {\prime}\right) \right\| _ {F} ^ {2} \\ = \| (W - J) \left[ \left(X _ {t - 1} - \bar {X} _ {t - 1} - \left(X _ {t - 1} ^ {\prime} - \bar {X} _ {t - 1} ^ {\prime}\right)\right) - \eta \left(Y _ {t - 1} - \bar {Y} _ {t - 1} - \left(Y _ {t - 1} ^ {\prime} - \bar {Y} _ {t - 1} ^ {\prime}\right)\right) \right] \| _ {F} ^ {2} \\ \leq \frac {1 + \lambda^ {2}}{2} \| X _ {t - 1} - \bar {X} _ {t - 1} - \left(X _ {t - 1} ^ {\prime} - \bar {X} _ {t - 1} ^ {\prime}\right) \| _ {F} ^ {2} + \frac {2 \eta^ {2} \lambda^ {2}}{1 - \lambda^ {2}} \| Y _ {t - 1} - \bar {Y} _ {t - 1} - \left(Y _ {t - 1} ^ {\prime} - \bar {Y} _ {t - 1} ^ {\prime}\right) \| _ {F} ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \leq \frac {2 \eta^ {2} \lambda^ {2}}{1 - \lambda^ {2}} \sum_ {\tau = s + 1} ^ {t - 1} \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - \tau - 1} \| Y _ {\tau} - \bar {Y} _ {\tau} - \left(Y _ {\tau} ^ {\prime} - \bar {Y} _ {\tau} ^ {\prime}\right) \| _ {F} ^ {2} \\ + \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - s - 1} \| X _ {s + 1} - \bar {X} _ {s + 1} - \left(X _ {s + 1} ^ {\prime} - \bar {X} _ {s + 1} ^ {\prime}\right) \| _ {F} ^ {2} \\ = \frac {2 \eta^ {2} \lambda^ {2}}{1 - \lambda} \sum_ {\tau = s + 1} ^ {t - 1} \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - \tau - 1} \| Y _ {\tau} - \bar {Y} _ {\tau} - \left(Y _ {\tau} ^ {\prime} - \bar {Y} _ {\tau} ^ {\prime}\right) \| _ {F} ^ {2} + \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - s - 1} \lambda^ {2} \left(n - n _ {s}\right) p _ {s} r _ {0} ^ {2} \tag {55} \\ \end{array}
$$
where we apply recursion in the second inequality and use the definition of the decoupled sequences in the last equality. Similarly, by recursion we also have
$$
\begin{array}{l} \left\| Y _ {t} - \bar {Y} _ {t} - \left(Y _ {t} ^ {\prime} - \bar {Y} _ {t} ^ {\prime}\right) \right\| _ {F} ^ {2} \\ \leq \frac {1 + \lambda^ {2}}{2} \| Y _ {t - 1} - \bar {Y} _ {t - 1} - (Y _ {t - 1} ^ {\prime} - \bar {Y} _ {t - 1} ^ {\prime}) \| _ {F} ^ {2} + \frac {\lambda^ {2} + \lambda^ {4}}{1 - \lambda^ {2}} \| V _ {t} - V _ {t - 1} - (V _ {t} ^ {\prime} - V _ {t - 1} ^ {\prime}) \| _ {F} ^ {2} \\ \leq \frac {2 \lambda^ {2}}{1 - \lambda} \sum_ {\tau = s + 1} ^ {t} \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - \tau} \| V _ {\tau} - V _ {\tau - 1} - \left(V _ {\tau} ^ {\prime} - V _ {\tau - 1} ^ {\prime}\right) \| _ {F} ^ {2} \tag {56} \\ \end{array}
$$
Combining above two inequalities, we achieve
$$
\begin{array}{l} \left\| X _ {t} - \bar {X} _ {t} - \left(X _ {t} ^ {\prime} - \bar {X} _ {t} ^ {\prime}\right) \right\| _ {F} ^ {2} \\ \leq \frac {2 \eta^ {2} \lambda^ {4}}{(1 - \lambda) ^ {2}} \sum_ {\tau = s + 1} ^ {t - 1} (\frac {1 + \lambda^ {2}}{2}) ^ {t - \tau - 1} (t - \tau) \| V _ {\tau} - V _ {\tau - 1} - (V _ {\tau} ^ {\prime} - V _ {\tau - 1} ^ {\prime}) \| _ {F} ^ {2} \\ + \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - s - 1} \lambda^ {2} \left(n - n _ {s}\right) p _ {s} r _ {0} ^ {2} \tag {57} \\ \end{array}
$$
According to the update rule of  $v_{t}^{(i)}$  we have
$$
\begin{array}{l} v _ {t} ^ {(i)} - v _ {t - 1} ^ {(i)} - \left(v _ {t} ^ {(i) ^ {\prime}} - v _ {t - 1} ^ {(i) ^ {\prime}}\right) - (1 - \beta) \left(v _ {t - 1} ^ {(i)} - v _ {t - 2} ^ {(i)} - \left(v _ {t - 1} ^ {(i) ^ {\prime}} - v _ {t - 2} ^ {(i) ^ {\prime}}\right)\right) \\ = \nabla F _ {i} (x _ {t} ^ {(i)}, \xi_ {t} ^ {(i)}) - (1 - \beta) \nabla F _ {i} (x _ {t - 1} ^ {(i)}, \xi_ {t} ^ {(i)}) - \nabla F _ {i} (x _ {t} ^ {(i) ^ {\prime}}, \xi_ {t} ^ {(i)}) + (1 - \beta) \nabla F _ {i} (x _ {t - 1} ^ {(i) ^ {\prime}}, \xi_ {t} ^ {(i)}) \\ - \left[ \nabla F _ {i} (x _ {t - 1} ^ {(i)}, \xi_ {t - 1} ^ {(i)}) - (1 - \beta) \nabla F _ {i} (x _ {t - 2} ^ {(i)}, \xi_ {t - 1} ^ {(i)}) - \nabla F _ {i} (x _ {t - 1} ^ {(i) ^ {\prime}}, \xi_ {t - 1} ^ {(i)}) \right. \\ + (1 - \beta) \nabla F _ {i} \left(x _ {t - 2} ^ {(i) ^ {\prime}}, \xi_ {t - 1} ^ {(i)}\right) ] \tag {58} \\ \end{array}
$$
Then mimic the estimation of  $\nu_{t}$ , we can obtain
$$
\begin{array}{l} \left\| V _ {t} - V _ {t - 1} - \left(V _ {t} ^ {\prime} - V _ {t - 1} ^ {\prime}\right) \right\| _ {F} ^ {2} \\ \leq \frac {3 2 \log (4 / \delta)}{b _ {1}} \sum_ {\tau = s} ^ {t - 1} \sum_ {i = 1} ^ {n} [ 2 L \| w _ {\tau + 1} ^ {(i)} - w _ {\tau} ^ {(i)} \| + (2 \beta L + 3 \rho C _ {d}) \| w _ {\tau} ^ {(i)} \| + 3 \rho C _ {d} \| w _ {\tau + 1} ^ {(i)} \| ] ^ {2} \\ + 4 L ^ {2} \sum_ {i = 1} ^ {n} \left\| w _ {t} ^ {(i)} - w _ {t - 1} ^ {(i)} \right\| ^ {2} + 3 6 \rho^ {2} C _ {d} ^ {2} \sum_ {i = 1} ^ {n} \left\| w _ {t} ^ {(i)} \right\| ^ {2} \tag {59} \\ \end{array}
$$
Combining above inequalities and the parameter setting of  $\beta$ , we can obtain
$$
\begin{array}{l} \frac {1}{n} \| X _ {t} - \bar {X} _ {t} - (X _ {t} ^ {\prime} - \bar {X} _ {t} ^ {\prime}) \| _ {F} ^ {2} - (\frac {1 + \lambda^ {2}}{2}) ^ {t - s - 1} \lambda^ {2} p _ {s} r _ {0} ^ {2} \\ \leq \left(\frac {2 0 0 0 \log (4 / \delta) \eta^ {2} \rho^ {2} C _ {d} ^ {2} (t - s) \lambda^ {4}}{(1 - \lambda) ^ {2} b _ {1}} + \frac {7 2 \eta^ {2} \rho^ {2} C _ {d} ^ {2} \lambda^ {4}}{(1 - \lambda) ^ {2}}\right) \sum_ {\tau = s + 1} ^ {t - 1} \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - \tau - 1} \frac {(t - \tau)}{n} \sum_ {i = 1} ^ {n} \| w _ {\tau} ^ {(i)} \| ^ {2} \\ + \left(\frac {5 0 0 \log (4 / \delta) L ^ {2} \eta^ {2} (t - s) \lambda^ {4}}{(1 - \lambda) ^ {2} b _ {1}} + \frac {8 L ^ {2} \eta^ {2} \lambda^ {4}}{(1 - \lambda) ^ {2}}\right) \sum_ {\tau = s + 1} ^ {t - 1} \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - \tau - 1} \frac {(t - \tau)}{n} \sum_ {i = 1} ^ {n} \| w _ {\tau} ^ {(i)} - w _ {\tau - 1} ^ {(i)} \| ^ {2} \\ \leq \frac {L ^ {2} \eta^ {2} \lambda^ {4}}{(1 - \lambda) ^ {2}} (3 2 + \frac {2 0 0 0 \log (4 / \delta) (t - s)}{b _ {1}}) \sum_ {\tau = s + 1} ^ {t - 1} (\frac {1 + \lambda^ {2}}{2}) ^ {t - \tau - 1} \frac {(t - \tau)}{n} \sum_ {i = 1} ^ {n} \| X _ {\tau} - \bar {X} _ {\tau} - (X _ {\tau} ^ {\prime} - \bar {X} _ {\tau} ^ {\prime}) \| _ {F} ^ {2} \\ + \left(\frac {2 0 0 0 \log (4 / \delta) \eta^ {2} \rho^ {2} C _ {d} ^ {2} (t - s) \lambda^ {4}}{(1 - \lambda) ^ {2} b _ {1}} + \frac {7 2 \eta^ {2} \rho^ {2} C _ {d} ^ {2} \lambda^ {4}}{(1 - \lambda) ^ {2}}\right) \sum_ {\tau = s + 1} ^ {t - 1} \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - \tau - 1} (t - \tau) \| w _ {\tau} \| ^ {2} \\ + \left(\frac {5 0 0 \log (4 / \delta) L ^ {2} \eta^ {2} (t - s) \lambda^ {4}}{(1 - \lambda) ^ {2} b _ {1}} + \frac {8 L ^ {2} \eta^ {2} \lambda^ {4}}{(1 - \lambda) ^ {2}}\right) \sum_ {\tau = s + 1} ^ {t - 1} \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - \tau - 1} (t - \tau) \| w _ {\tau} - w _ {\tau - 1} \| ^ {2} \tag {60} \\ \end{array}
$$
Using Eq. (36), Eq. (47) and Eq. (54) we have
$$
\frac {L ^ {2}}{n} \left\| X _ {t} - \bar {X} _ {t} - \left(X _ {t} ^ {\prime} - \bar {X} _ {t} ^ {\prime}\right) \right\| _ {F} ^ {2}
$$
$$
\begin{array}{l} \leq B _ {1} t ^ {2} \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - s - 1} L ^ {2} p _ {s} r _ {0} ^ {2} + B _ {2} (1 + \eta \gamma) ^ {2 (t - s - 1)} L ^ {2} p _ {s} ^ {2} r _ {0} ^ {2} \sum_ {\tau = s + 1} ^ {t - 1} \left(\frac {(1 + \lambda^ {2}) (1 + \eta \gamma) ^ {2}}{2}\right) ^ {t - \tau - 1} (t - \tau) \\ + B _ {3} (1 + \eta \gamma) ^ {2 (t - s - 1)} L ^ {2} p _ {s} ^ {2} r _ {0} ^ {2} \sum_ {\tau = s + 1} ^ {t - 1} \left(\frac {(1 + \lambda^ {2}) (1 + \eta \gamma) ^ {2}}{2}\right) ^ {t - \tau - 1} \frac {t - \tau}{(\tau - s) ^ {2}} \\ + \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - s - 1} L ^ {2} p _ {s} r _ {0} ^ {2} \tag {61} \\ \end{array}
$$
$$
B _ {1} = \frac {4 L ^ {2} \eta^ {2}}{(1 - \lambda) ^ {2}} (3 2 + \frac {2 0 0 0 \log (4 / \delta) (t - s)}{b _ {1}}) + 3 8 4 L ^ {2} \eta^ {2} (\frac {5 0 0 \log (4 / \delta) L ^ {2} \eta^ {2} (t - s)}{(1 - \lambda) ^ {2} b _ {1}} + \frac {8 L ^ {2} \eta^ {2}}{(1 - \lambda) ^ {2}})
$$
$$
\begin{array}{l} B _ {2} = \frac {1}{7 2 (1 - \lambda) ^ {2} C _ {T} ^ {2}} \left(2 + \frac {1 2 5 \log (4 / \delta) (t - s)}{b _ {1}}\right) + \frac {4 5 0 0 \log (4 / \delta) \eta^ {2} \rho^ {2} C _ {d} ^ {2} (t - s)}{(1 - \lambda) ^ {2} b _ {1}} + \frac {1 6 2 \eta^ {2} \rho^ {2} C _ {d} ^ {2}}{(1 - \lambda) ^ {2}} \\ + (8 L ^ {2} \eta^ {2} \gamma^ {2} + 5 4 L ^ {2} \eta^ {2} \rho^ {2} C _ {d} ^ {2} + \frac {1}{6 C _ {T} ^ {2}}) (\frac {5 0 0 \log (4 / \delta) L ^ {2} \eta^ {2} (t - s)}{(1 - \lambda) ^ {2} b _ {1}} + \frac {8 L ^ {2} \eta^ {2}}{(1 - \lambda) ^ {2}}) \\ \end{array}
$$
$$
\begin{array}{l} B _ {3} = \frac {4 8 \log (4 / \delta) L ^ {2} \eta^ {2}}{b _ {1}} \left(\frac {5 0 0 \log (4 / \delta) L ^ {2} \eta^ {2} (t - s)}{(1 - \lambda) ^ {2} b _ {1}} + \frac {8 L ^ {2} \eta^ {2}}{(1 - \lambda) ^ {2}}\right) \\ + \frac {L ^ {4} \eta^ {4} (1 + \eta \gamma) ^ {2 (t - s - 1)}}{(1 - \lambda) ^ {2} b _ {1}} (3 2 + \frac {2 0 0 0 \log (4 / \delta) (t - s)}{b _ {1}}) \tag {62} \\ \end{array}
$$
If  $t \geq \tilde{O} \left( \frac{1}{1 - \lambda} \right)$ ,  $t^2 \left( \frac{1 + \lambda^2}{2} \right)^{t - s - 1}$  is small and the first term of Eq. (61) can be merged to the second term. Otherwise if  $t < \tilde{O} \left( \frac{1}{1 - \lambda} \right)$ , it can be merged to the last term according to the parameter setting of  $\eta$  and  $b_1$ . When  $\epsilon$  is small, we have  $\frac{(1 + \lambda^2)(1 + \eta\gamma)^2}{2} \leq \frac{3 + \lambda^2}{4}$ . Hence the second term of Eq. (61) can be bounded by Lemma 8. The third term of Eq. (61) can be estimated by Lemma 9 (the case of  $t < \tilde{O} \left( \frac{1}{1 - \lambda} \right)$  can be addressed by the parameter setting of  $\eta$  and  $b_1$ ). Therefore, we can prove
$$
\begin{array}{l} \frac {L ^ {2}}{n} \| X _ {t} - \bar {X} _ {t} - \left(X _ {t} ^ {\prime} - \bar {X} _ {t} ^ {\prime}\right) \| _ {F} ^ {2} \leq 4 \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {t - s - 1} L ^ {2} p _ {s} r _ {0} ^ {2} + \frac {1}{2 3 0 4 \eta C _ {T}} (1 + \eta \gamma) ^ {2 (t - s - 1)} p _ {s} ^ {2} r _ {0} ^ {2} \\ + \frac {L ^ {2} \eta^ {2} (1 + \eta \gamma) ^ {2 (t - s - 1)} L ^ {2} p _ {s} ^ {2} r _ {0} ^ {2}}{1 6 b _ {1} (t - s) ^ {2}} \tag {63} \\ \end{array}
$$
because of the parameter setting. We should notice that here we also use the relation  $\frac{\eta}{b_1\gamma} \leq O(1)$ , which is always satisfied according to the setting of  $b_1$ . Based on Eq. (53) and Eq. (63), it is easy to check that  $\zeta_t$  satisfies Eq. (40). Moreover, we have
$$
\begin{array}{l} \eta \sum_ {\tau = s + 1} ^ {t - 1} (1 + \eta \gamma) ^ {t - \tau - 1} \| \zeta_ {\tau} \| \\ \leq L \eta (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \left(8 \sum_ {\tau = s + 1} ^ {t - 1} \left(\frac {4 + \lambda^ {2}}{5}\right) ^ {t - s - 1} + \sum_ {\tau = s + 1} ^ {t - 1} \frac {1}{\tau - s}\right) + \frac {1}{1 2} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \\ \leq L \eta \left(\frac {8 0}{1 - \lambda} + \log \left(C _ {T}\right)\right) \left(1 + \eta \gamma\right) ^ {t - s - 1} p _ {s} r _ {0} + \frac {1}{1 2} \left(1 + \eta \gamma\right) ^ {t - s - 1} p _ {s} r _ {0} \\ \leq \frac {1}{6} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \tag {64} \\ \end{array}
$$
where the first inequality is derived by
$$
\frac {1 + \lambda^ {2}}{2} \leq \left(\frac {3 + \lambda^ {2}}{4}\right) ^ {2} \quad \text {a n d} \quad \frac {(3 + \lambda^ {2}) (1 + \eta \gamma)}{4} \leq \frac {4 + \lambda^ {2}}{5} \tag {65}
$$
Now combining Eq. (41), Eq. (51) and Eq. (64), we can reach the conclusion in Eq. (38) and finish the proof of the induction. Recall the assumption at the beginning, we have
$$
\frac {1}{2} (1 + \eta \gamma) ^ {C _ {T}} p _ {s} r _ {0} \leq w _ {C _ {T}} \leq 2 \bar {d} \leq 6 C _ {d} \tag {66}
$$
since  $\| \bar{x}_t - \bar{x}_t' \| \leq \| \bar{x}_t - \bar{x}_s \| + \| \bar{x}_t' - \bar{x}_s \|$ . Eq. (66) implies that
$$
C _ {T} \leq \frac {\log \left(1 2 C _ {d} / \left(p _ {s} r _ {0}\right)\right)}{\log (1 + \eta \gamma)} <   \frac {2 \log \left(1 2 n C _ {d} / r _ {0}\right)}{\eta \gamma} \tag {67}
$$
which conflicts with the definition of  $C_T$ . Therefore, the proof of Lemma 6 is finished.
C.7 Proof of Lemma 7
Proof. If node  $i$  enters the escaping phase in iteration  $s'$  before iteration  $s$  and does not break it in iteration  $s + C_T$ , then for  $s \leq t \leq s + C_T$ , we have  $\|x_t^{(i)} - x_s^{(i)}\| \leq \|x_t^{(i)} - x_{s'}^{(i)}\| + \|x_s^{(i)} - x_{s'}^{(i)}\| \leq 2C_d$ . Therefore, there are at least  $\frac{n}{10}$  worker nodes satisfying  $\max_{s \leq t \leq s + C_T} \|x_t^{(i)} - x_s^{(i)}\| \leq 2C_d$ .
Suppose  $\min eig(\nabla^2 f(\bar{x}_s)) \leq -\epsilon_H$  and  $\mathbf{e_1}$  is the corresponding eigenvector. Let  $S_{i}$  denote the region of the perturbation on node  $i$  that PEDESTAL will terminate in iteration  $s + C_T$ , i.e.,  $\frac{n}{10}$  workers will not break the escaping phase. Then by Lemma 6 we can conclude that there must exist one worker node such that the projection of  $S_{i}$  onto direction  $\mathbf{e_1}$  is smaller than  $r_0$ . Since the perturbation  $\xi_{i}$  is drawn from uniform distribution, the probability of  $\xi_{i} \in S_{i}$  can be bounded by
$$
P r \left(\xi_ {i} \in \mathcal {S} _ {i}\right) \leq \frac {r _ {0} V (B a l l (d - 1 , r))}{V (B a l l (d , r))} \leq \delta \tag {68}
$$
where  $V(\cdot)$  denotes the volume and  $\text{Ball}(d, r)$  denotes the  $d$ -dimensional ball with radius  $r$ . The last inequality is achieved by the definition of  $r_0$ . Therefore, we can prove that  $\bar{x}_s$  is a second-order stationary point with probability at least  $1 - \delta$ .
D Additional Theoretical Result
In this section we will provide some additional theoretical result of our PEDESTAL algorithm. First we will demonstrate the convergence analysis of the case  $\epsilon_H < \sqrt{\epsilon}$ , i.e.,  $\alpha > 0.5$ . Next, we will discuss the strategy of using fixed number of iterations in each descent and escaping phase, which motivates the design of PEDESTAL.
D.1 Smaller Tolerance for Second-Order Optimality
When  $\epsilon_H < \sqrt{\epsilon}$ , the conclusions of previous Lemmas are still satisfied except Lemma 4. In this case,  $C_d = C_2\eta C_T\epsilon^\mu$  where  $\mu = 2\alpha > 1$ . Parameter  $C_d$  should be smaller than the original setting in Lemma 4, which results in more iterations to converge. Fortunately, the analysis of Lemma 4 can be adjusted and we can achieve Theorem 2. The proof is provided as follows.
Proof. The fourth term of  $\mathcal{D}_t$  in Lemma 3 is derived by  $\frac{\eta\beta\sigma^2}{b_1}$  and at this time we will set  $b_{1} \geq \epsilon^{-(2\mu -1 - \theta)}$  so that the  $\epsilon$  term is replaced by  $\epsilon^{\mu}$ . The last term of  $\mathcal{D}_t$  can be written as
$$
\sum_ {t = 0} ^ {T - 1} \frac {7 p _ {t} \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{4 \eta} = \frac {1}{n} \sum_ {(t, i) \in \mathcal {P}} \frac {7 \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{4 \eta} \tag {69}
$$