SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathcal{P}$  is the set of all pairs of  $(t,i)$  such that node  $i$  draws perturbation in iteration  $t$ . We can divide  $\mathcal{P}$  into two parts.  $\mathcal{P}_1$  contains all pairs of  $(t,i)$  such that node  $i$  breaks the escaping phase within  $M$  iterations, where  $M$  is an integer to be decided later. The rest part is denoted by  $\mathcal{P}_2$ .
For any  $(t,i)\in \mathcal{P}_1$ , suppose node  $i$  breaks escaping phase in iteration  $t + m$ , where  $m\leq M$ . Then node  $i$  will never draw perturbation between iteration  $t$  and iteration  $t + M$ . Mimic the steps of Eq. (26), by Cauchy-Schwartz inequality we can obtain
$$
\sum_ {\tau = t} ^ {t + m} \left\| x _ {\tau + 1} ^ {(i)} - x _ {\tau} ^ {(i)} \right\| ^ {2} \geq \frac {C _ {d} ^ {2}}{M} \tag {70}
$$
Let  $M = \epsilon^{-2 - 2\theta +2\alpha}$  . Then we have
$$
\frac {(1 - \lambda) ^ {2}}{5 1 2 \eta} \sum_ {\tau = t} ^ {t + m} \| x _ {\tau + 1} ^ {(i)} - x _ {\tau} ^ {(i)} \| ^ {2} \geq \frac {7 \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{4 \eta} \tag {71}
$$
$$
\frac {(1 - \lambda) ^ {2}}{5 1 2 n \eta} \sum_ {\tau = t} ^ {t + m} \sum_ {i = 1} ^ {n} \| x _ {\tau + 1} ^ {(i)} - x _ {\tau} ^ {(i)} \| ^ {2} \geq \frac {1}{n} \sum_ {(t, i) \in \mathcal {P} _ {1}} \frac {7 \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{4 \eta} \tag {72}
$$
by the parameter setting of  $C_v$ . On the other hand, if  $(t,i) \in \mathcal{P}_2$ , then node  $i$  will not break the escaping phase in  $M$  steps and hence the perturbation step will not execute, either. Therefore, we have estimation
$$
\frac {1}{n} \sum_ {(t, i) \in \mathcal {P} _ {2}} \frac {7 \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{4 \eta} \leq \sum_ {t = 0} ^ {T - 1} \frac {7 \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{4 M \eta} \tag {73}
$$
With Eq. (72), Eq. (73) and the new setting of  $b_{1}$ , the descent in Lemma 3 can be improved to
$$
\mathcal {D} _ {t} = \frac {1}{1 6 \eta} \omega_ {t} + \frac {(1 - \lambda) ^ {2}}{5 1 2 n \eta} \sum_ {i = 1} ^ {n} \| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \| ^ {2} + \frac {\eta}{2 n} \sum_ {i = 1} ^ {n} \| y _ {t} ^ {(i)} \| ^ {2} - \frac {2 0 0 \eta \epsilon^ {2 \mu} \sigma^ {2}}{(1 - \lambda) ^ {2} C _ {1} ^ {2}} - \frac {7 (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{4 M \eta}
$$
When  $\theta \geq 3\alpha - 2$ , we have  $\frac{\epsilon^2}{M} \leq \epsilon^{2\mu}$  and Lemma 4 still holds but the conclusion is changed to
$$
\sum_ {t \in \mathcal {I}} \mathcal {D} _ {t} \geq | \mathcal {I} | \cdot \frac {(1 - \lambda) ^ {2} C _ {2} ^ {2} \eta \epsilon^ {2 \mu}}{1 0 0 0 0}
$$
In this case, PEDESTAL algorithm will terminate in  $\tilde{O} (\epsilon^{-\theta -2\mu})$  iterations. In Lemma 6 and Lemma 7 we need the relations
$$
\frac {\eta^ {2} C _ {d} ^ {2} C _ {T} ^ {3}}{b _ {1}} \leq O (1), \quad \frac {\eta}{b _ {1} \epsilon_ {H}} \leq O (1) \tag {74}
$$
which implies  $b_{1} \geq \tilde{O}(\epsilon^{-\theta - \alpha})$ . Therefore, we set  $b_{1} = \tilde{\Theta}(\epsilon^{-\max \{4\alpha - 1 - \theta, \theta + \alpha\}})$  with the condition  $\theta \geq 3\alpha - 2$ . When  $\alpha \leq 1$ , we set  $\theta = \frac{3\alpha - 1}{2}$ , which satisfies  $\theta \geq 3\alpha - 2$  and
$$
4 \alpha - 1 - \theta = \theta + \alpha = \frac {5 \alpha - 1}{2} \tag {75}
$$
The gradient complexity in this case is
$$
\tilde {O} \left(\epsilon^ {- \frac {1 1 \alpha - 1}{2}} \cdot \epsilon^ {- \frac {5 \alpha - 1}{2}}\right) = \tilde {O} \left(\epsilon^ {- 8 \alpha + 1}\right) \tag {76}
$$
When  $\alpha > 1$ , we have  $\theta = 3\alpha - 2$  and  $b_{1} = \tilde{\Theta}(\epsilon^{-(4\alpha - 2)})$ . The gradient complexity is
$$
\tilde {O} \left(\epsilon^ {- (7 \alpha - 2)} \cdot \epsilon^ {- (4 \alpha - 2)}\right) = \tilde {O} \left(\epsilon^ {- 1 1 \alpha + 4}\right) \tag {77}
$$
which finishes the proof of Theorem 2.
Therefore, the gradient complexity over all cases of  $\alpha$  can be written by
$$
\tilde {O} \left(\epsilon^ {- 3} + \epsilon \epsilon_ {H} ^ {- 8} + \epsilon^ {4} \epsilon_ {H} ^ {- 1 1}\right) \tag {78}
$$
D.2 Phases with Fixed Number of Iterations
If a decentralized stochastic perturbed gradient descent method adopts the strategy of fixed number of iterations in each phase, the gradient complexity in the descent phase should be at least  $O(\epsilon^{-3})$  to ensure the first-order stationary point. But the total descent of a descent phase could be small because it is possible that it is stuck at a saddle point after only a few steps. Hence we need to consider the descent in the escaping phase. According to Lemma 3 and Lemma 4 we can see the descent of an escaping phase is  $O\left(\frac{C_d^2}{\eta C_T}\right)$ . As the conditions  $\eta C_d C_T \leq O(1)$  and  $C_T = \tilde{O}\left(\frac{1}{\eta \epsilon_H}\right)$  are required in Lemma 6, we can obtain that the total descent of an escaping phase is no larger than  $\tilde{O}(\epsilon_H^3)$ . In the classic setting of  $\epsilon_H = \sqrt{\epsilon}$ , the total descent of an escaping is upper bounded by  $\tilde{O}(\epsilon^{1.5})$ . Consequently, the total gradient complexity to achieve  $(\epsilon, \sqrt{\epsilon})$ -second-order stationary point is at least  $\tilde{O}(\epsilon^{-4.5})$ , which is worse than the result of our PEDESTAL.
E Auxiliary Lemmas
Lemma 8. Let  $0 < a < 1$ . Then we have
$$
\sum_ {\tau = 1} ^ {t} \tau a ^ {\tau - 1} = \frac {1 - a ^ {t}}{(1 - a) ^ {2}} - \frac {t a ^ {t}}{1 - a}
$$
Lemma 9. Let  $0 < a < 1$ . When  $t \geq \tilde{O}\left(\frac{1}{1 - a}\right)$ , we have
$$
\sum_ {\tau = 1} ^ {t} \frac {\tau a ^ {\tau - 1}}{(t + 1 - \tau) ^ {2}} \leq \frac {8}{t ^ {2} (1 - a) ^ {2}}
$$
Proof. When  $\tau \leq \frac{t}{2}$ , by Lemma 8 we have
$$
\sum_ {\tau \leq t / 2} \frac {\tau a ^ {\tau - 1}}{(t + 1 - \tau) ^ {2}} \leq \frac {4}{t ^ {2} (1 - a) ^ {2}} \tag {79}
$$
When  $\tau >\frac{t}{2}$  , we have
$$
\sum_ {\tau > t / 2} \frac {\tau a ^ {\tau - 1}}{(t + 1 - \tau) ^ {2}} \leq \sum_ {\tau > t / 2} \tau a ^ {\tau - 1} \leq a ^ {t / 2} \left(\frac {t}{2 (1 - a)} + \frac {1}{(1 - a) ^ {2}}\right) \tag {80}
$$
Therefore, we can reach the conclusion when  $t \geq \tilde{O}\left(\frac{1}{1 - a}\right)$ .
Lemma 10. (Definition of Variance) For any random variable  $X$ , we have
$$
\mathbb {E} [ X - \mathbb {E} X ] ^ {2} = \mathbb {E} X ^ {2} - (\mathbb {E} X) ^ {2}
$$
Lemma 11. (Lemma D.1 in (Chen et al. [2022])) Let  $\epsilon_{1:k} \in \mathbb{R}^d$  be a vector-valued martingale difference sequence with respect to  $\mathcal{F}_k$ , i.e., for each  $k \in [K]$ ,  $\mathbb{E}[\epsilon_k|\mathcal{F}_k] = 0$  and  $\| \epsilon_k \| \leq B_k$ , then with probability  $1 - \delta$  we have
$$
\left\| \sum_ {k = 1} ^ {K} \epsilon_ {k} \right\| ^ {2} \leq 4 \log (4 / \delta) \sum_ {k = 1} ^ {K} B _ {k} ^ {2} \tag {81}
$$