SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(f)  $d = 100$  , exponential
Figure 3: Experimental results of the decentralized matrix sensing task on different network topology for  $d = 50$  and  $d = 100$ . Data is assigned to worker nodes by Dirichlet distribution. The y-axis is the loss function value and the x-axis is the number of gradient oracles divided by the number of data  $N$ .
D-PSGD	GTDSGD	D-GET	D-SPIDER-SFO	GTHSGD	PEDESTAL
d=50, ring	-0.0332	-0.0327	-0.0333	-0.0328	-0.0329	-1.78e-5
d=50, toroidal	-0.0331	-0.0334	-0.0334	-0.0327	-0.0329	-4.18e-5
d=50, exponential	-0.0323	-0.0330	-0.0331	-0.0332	-0.0333	-1.09e-6
d=100, ring	-0.0184	-0.0184	-0.0184	-0.0184	-0.0185	-2.07e-6
d=100, toroidal	-0.0185	-0.0186	-0.0185	-0.0184	-0.0184	-2.25e-7
d=100, exponential	-0.0184	-0.0184	-0.0186	-0.0184	-0.0184	-3.07e-5
Table 2: Smallest eigenvalue of hessian matrix at the converged point (random data distribution).
D-PSGD	GTDSGD	D-GET	D-SPIDER-SFO	GTHSGD	PEDESTAL
d=50, ring	-0.0332	-0.0337	-0.0332	-0.0325	-0.0330	-3.60e-6
d=50, toroidal	-0.0334	-0.0324	-0.0329	-0.0325	-0.0327	-2.29e-5
d=50, exponential	-0.0334	-0.0326	-0.0333	-0.0330	-0.0328	-3.97e-5
d=100, ring	-0.0184	-0.0184	-0.0184	-0.0185	-0.0183	-4.48e-5
d=100, toroidal	-0.0184	-0.0184	-0.0184	-0.0184	-0.0185	-1.24e-5
d=100, exponential	-0.0186	-0.0185	-0.0186	-0.0183	-0.0185	-3.63e-6
Table 3: Smallest eigenvalue of hessian matrix at the converged point (Dirichlet data distribution).
We define matrix  $X_{t} = [x_{t}^{(1)},\dots ,x_{t}^{(n)}]\in \mathbb{R}^{d\times n}$  where  $x_{t}^{(i)}$  is the model parameter on  $i$ -th worker node with dimension  $d$  and  $n$  is the number of worker nodes. Similarly we have  $Y_{t} = [y_{t}^{(1)},\dots ,y_{t}^{(n)}]$ ,
$Z_{t} = [z_{t}^{(1)},\dots ,z_{t}^{(n)}]$  and  $V_{t} = [v_{t}^{(1)},\dots ,v_{t}^{(n)}]$ . Let  $\omega_{t} = \| \bar{x}_{t + 1} - \bar{x}_{t}\|^{2}$  and  $\Omega_t = Z_t - X_t$ . Define  $p_t = n_t / n$  where  $n_t$  is the number of worker nodes drawing perturbation in iteration  $t$ .
B.2 Outline
In this section we will provide the proof outline of Theorem 1. First, we prove some basic lemmas to estimate gradient noise and consensus error, which will be used frequently in later proof. The gradient noise is estimated by Lemma 1, the proof of which can be found in Section C.1. The consensus error is estimated by Lemma 2, the proof of which can be found in Section C.2.
Lemma 1. (Gradient Noise) Under Assumption 2 and Assumption 3 we have
(a)  $\frac{1}{nT}\sum_{t=0}^{T-1}\sum_{i=1}^{n}\|v_t^{(i)}-\nabla f_i(x_t^{(i)})\|^2 \leq \frac{16\log(4/\delta)\beta\sigma^2}{b_1}+\frac{384\log(4/\delta)L^2}{nb_1\beta T}\sum_{t=0}^{T-1}\|X_t-\bar{X}_t\|_F^2$
$$
+ \frac {1 9 2 \log (4 / \delta) L ^ {2}}{b _ {1} \beta T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} + \frac {2 \log (4 / \delta) \sigma^ {2}}{\beta b _ {0} T}
$$
(b)  $\frac{1}{T}\sum_{t = 1}^{T}\| \bar{v}_t - \frac{1}{n}\sum_{i = 1}^n\nabla f_i(x_t^{(i)})\| ^2\leq \frac{16\log(4 / \delta)\beta\sigma^2}{nb_1} +\frac{384\log(4 / \delta)L^2}{n^2b_1\beta T}\sum_{t = 1}^{T}\| X_t - \bar{X}_t\| _F^2$
$$
+ \frac {1 9 2 \log (4 / \delta) L ^ {2}}{n b _ {1} \beta T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} + \frac {2 \log (4 / \delta) \sigma^ {2}}{n \beta b _ {0} T}
$$
Lemma 2. (Consensus Error) Let  $\eta \leq \frac{(1 - \lambda)^{2}\epsilon^{\theta}}{600\log(4 / \delta)\lambda^{2}L}$ ,  $\beta = C_1^{-1}\epsilon^{1 + \theta}$  and  $b_{1}\geq C_{1}\epsilon^{-1 + \theta}$  where  $C_1\geq 1$  is a constant. Under Assumption 2, 3 and 5 we have
(a)  $\frac{1}{T}\sum_{t = 1}^{T}\| X_t - \bar{X}_t\| _F^2\leq \frac{160000n\log(4 / \delta)L^2\eta^2\lambda^4}{(1 - \lambda)^4\min\{b_1\beta,1\}T}\sum_{t = 0}^{T - 1}\omega_t + \frac{12288n\log(4 / \delta)\beta\eta^2\lambda^4\sigma^2}{(1 - \lambda)^4b_1}$
$$
+ \frac {2 0 0 0 n \log (4 / \delta) \eta^ {2} \lambda^ {4} \sigma^ {2}}{(1 - \lambda) ^ {4} \beta b _ {0} T} + \frac {1 2 8 \lambda^ {4} \eta^ {2}}{(1 - \lambda) ^ {3} T} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} + \sum_ {t = 0} ^ {T - 1} \frac {6 4 n \lambda^ {2} p _ {t} (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{(1 - \lambda) ^ {2} T}
$$
(b)  $\frac{1}{T}\sum_{t = 0}^{T - 1}\| Y_t - \bar{Y}_t\| _F^2\leq \frac{4644\log(4 / \delta)nL^2\lambda^2}{(1 - \lambda)\min\{b_1\beta,1\}T}\sum_{t = 0}^{T - 1}\omega_t + \frac{384\log(4 / \delta)n\lambda^2\beta\sigma^2}{(1 - \lambda)b_1}$
$$
+ \frac {5 0 \log (4 / \delta) n \lambda^ {2} \sigma^ {2}}{(1 - \lambda) \beta b _ {0} T} + \frac {8 \lambda^ {2}}{(1 - \lambda) T} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} + \sum_ {t = 0} ^ {T - 1} \frac {1 5 0 0 0 0 \log (4 / \delta) n L ^ {2} \lambda^ {4} p _ {t} (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{(1 - \lambda) ^ {3} \min \{b _ {1} \beta , 1 \} T}
$$
Next we will prove that PEDESTAL will terminate in certain number of iterations. Under Assumption 2, 3 and 5, we can prove the following Lemma 3. The proof is demonstrated in Section C.3.
Lemma 3. (Descent) Let  $\eta \leq \frac{(1 - \lambda)^2\epsilon^\theta}{600\log(4 / \delta)\lambda^2L}$ ,  $\beta = C_1^{-1}\epsilon^{1 + \theta}$ ,  $b_{1} \geq C_{1}\epsilon^{-1 + \theta}$  and  $b_{0} = C_{1}\epsilon^{-1}$  where  $C_1 \geq 1$  is a constant. Under Assumption 2, 3 and 5 we have
$$
f (\bar {x} _ {T}) \leq f (x _ {0}) + \frac {\sigma^ {2}}{L} + \frac {1}{n L} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} - \sum_ {t = 0} ^ {T - 1} \mathcal {D} _ {t}
$$
$$
\mathcal {D} _ {t} = \frac {1}{1 6 \eta} \omega_ {t} + \frac {(1 - \lambda) ^ {2}}{2 5 6 n \eta} \sum_ {i = 1} ^ {n} \| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \| ^ {2} + \frac {\eta}{2 n} \sum_ {i = 1} ^ {n} \| y _ {t} ^ {(i)} \| ^ {2} - \frac {2 0 0 \eta \epsilon^ {2} \sigma^ {2}}{(1 - \lambda) ^ {2} C _ {1} ^ {2}} - \frac {7 p _ {t} (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{4 \eta}
$$
Here we call  $\mathcal{D}_t$  the descent of iteration  $t$ . We categorize all iterations into three types:
type-A:  $p_t \geq \frac{1}{5}$ , type-B:  $p_t < \frac{1}{5}$  and  $\frac{1}{n} \sum_{i=1}^{n} \| y_t^{(i)} \|^2 \geq \frac{4C_v^2}{5}$ , type-C: otherwise
When at least  $\frac{n}{5}$  nodes drawing perturbation in iteration  $t$ , then it is type-A. There are two cases where  $p_t$  is small: most nodes in the descent phase or most nodes in the escaping phase. An iteration is type-B if  $p_t < \frac{1}{5}$  and  $\frac{1}{n}\sum_{i=1}^{n}\|y_t^{(i)}\|^2 \geq \frac{4C_v^2}{5}$ , which represents the case where most nodes are in the descent phase. And type-C iteration represents the case where most nodes are in the escaping phase. Next we will estimate type-A and type-C iteration with the following Lemma 4.
Lemma 4. Let  $\eta \leq \frac{(1 - \lambda)^2\epsilon^\theta}{600\log(4 / \delta)\lambda^2L}$ ,  $\beta = C_1^{-1}\epsilon^{1 + \theta}$ ,  $b_{1} \geq C_{1}\epsilon^{-1 + \theta}$ ,  $b_{0} = C_{1}\epsilon^{-1}$ ,  $C_d = C_2\eta C_T\epsilon$ ,  $C_v = \frac{(1 - \lambda)C_2\epsilon}{200}$  and  $r \leq \eta C_v / 4$  where  $C_1 = \frac{20000\sigma}{(1 - \lambda)^2C_2}$  and  $C_2$  is a constant. Under Assumption 2, 3 and 5, we can find disjoint intervals  $\mathcal{I} = \mathcal{I}_1 \cup \dots \cup \mathcal{I}_k$  such that the indexes of all type-A and type-C iterations except the last  $C_T$  iterations are contained in  $\mathcal{I}$  and the descent over  $\mathcal{I}$  can be estimated by
$$
\sum_ {t \in \mathcal {I}} \mathcal {D} _ {t} \geq | \mathcal {I} | \cdot \frac {(1 - \lambda) ^ {2} C _ {2} ^ {2} \eta \epsilon^ {2}}{1 0 0 0 0}
$$
where  $|\cdot|$  denotes the total number of the set.
Besides, for all type-B iteration  $t$ , we have the following estimation
Lemma 5. Let parameter and assumption settings be the same as Lemma 4, then for all type-B iteration  $t$  we have
$$
\mathcal {D} _ {t} \geq \frac {(1 - \lambda) ^ {2} C _ {2} ^ {2} \eta \epsilon^ {2}}{8 0 0 0 0 0 0}
$$
With Lemma 4, Lemma 5 and Assumption 1, we can conclude that PEDESTAL will terminate in  $\tilde{O} (\epsilon^{-2 - \theta}) + C_T$  iterations. As the last two negative terms in  $\mathcal{D}_t$  are canceled by  $\frac{1}{n}\sum_{i = 1}^{n}\| x_{t + 1}^{(i)} - x_t^{(i)}\|^2$  and  $\frac{1}{n}\sum_{i = 1}^{n}\| y_t^{(i)}\|^2$  respectively in Lemma 4 and Lemma 5, we have  $\frac{1}{\eta}\sum_{t = 0}^{T - 1}\omega_t\leq O(1)$ . Hence by Lemma 2 we know the consensus error  $\frac{1}{n}\| X_t - \bar{X}_t\| _F^2$  can be bounded by  $O(\epsilon^{1 + \theta})$  on average. Besides, from the parameter setting we can see  $C_v$  is  $\Theta (\epsilon)$ , which ensures the first-order optimality of the decentralized algorithm.
Finally, we will prove PEDESTAL is able to achieve second-order stationary point. First, we will give the small stuck region lemma in decentralized setting. Recall that  $\epsilon_H = \epsilon^\alpha$  is the tolerance of second-order stationary point. The proof is in Section C.6.
Lemma 6. (Small Stuck Region) Suppose  $n_s$  worker nodes draw perturbation in iteration  $s$  and  $-\gamma = \min eig(\nabla^2 f(\bar{x}_s)) \leq -\epsilon_H$ . Let  $\eta \leq \frac{(1 - \lambda)^2\epsilon^\theta}{1000\sqrt{n}\log(C_T)\log(4 / \delta)\lambda^2L}$ ,  $\beta = C_1^{-1}\epsilon^{1 + \theta}$ ,  $b_1 \geq 1000C_1\epsilon^{2 - \theta -5\alpha}$ ,  $C_d = C_2\eta C_T\epsilon^\mu$  and  $C_T = \log(12nC_d / r_0) / (\eta \gamma)$  where  $C_1 = \frac{20000}{(1 - \lambda)^2C_2}$ ,  $C_2 \leq \frac{1 - \lambda}{2000\log(4 / \delta)\rho\log(C_d)}$  and  $\mu = \max \{1, 2\alpha\}$ . Let  $X_t$  and  $X_t'$  be two coupled decentralized sequences by running PEDESTAL from  $X_s$  with  $X_s = X_s'$ ,  $x_{s+1}^{(i)} = x_{s+1}^{(i)'}$  if node  $i$  does not draw perturbation in iteration  $s$  and  $x_{s+1}^{(i)} = x_{s+1}^{(i)'} + r_0\mathbf{e}_1$  otherwise. Here  $\mathbf{e}_1$  is the eigenvector with respect to the smallest eigenvalue  $\gamma$ . Define  $d_i = \max_{s \leq t \leq s + C_T} \{\|x_t^{(i)} - x_s^{(i)}\|, \|x_t^{(i)'} - x_s^{(i)}\|\}$ . Then there are at least  $\frac{9n}{10}$  nodes such that  $d_i \geq 2C_d$ .
In decentralized small stuck region lemma, the consensus error will lead to a new term (see Eq. (34)) and make the proof more complicated. In our proof, we use the condition of  $\epsilon_H \geq \epsilon$ , i.e.,  $\alpha \leq 1$ . For smaller  $\epsilon_H$  the batchsize  $b_1$  is required to set larger. With Lemma 6, we can prove that when PEDESTAL is terminated, it finds a local minimum with high probability.
Lemma 7. Let  $r_0 = \delta r / \sqrt{d}$  where  $d$  is the dimension of model parameter. Other parameters are the same as Lemma 6. Suppose PEDESTAL is terminated in iteration  $s + C_T$ . Then  $\bar{x}_s$  is a second-order stationary point with probability at least  $1 - \delta$ .
Lemma 7 provides the guarantee of second-order optimality of PEDESTAL. When  $\epsilon_H \geq \sqrt{\epsilon}$ , i.e.,  $\alpha \leq 0.5$  (including the classic setting  $\epsilon_H = \sqrt{\epsilon}$ ), the parameter settings of all lemmas are consistent and the main theorem is proven. The total gradient complexity is
$$
\tilde {O} \left(\epsilon^ {- 2 - \theta} \cdot \epsilon^ {- 1 + \theta}\right) = \tilde {O} \left(\epsilon^ {- 3}\right)
$$
When  $\alpha = 0.5$ , we have  $\theta = 0.25$  and  $b_{1} = \Theta (\epsilon^{-0.75})$ . When  $\alpha \leq 0.2$ , we can set  $\theta = 1$  and  $b_{1} = O(1)$ , which is result of PEDESTAL-S. In Section D we will provide the analysis of the case  $\alpha > 0.5$  with a different parameter setting of  $\theta$  and  $b_{1}$ . We can achieve the gradient complexity of
$$
\tilde {O} \left(\epsilon^ {- 3} + \epsilon \epsilon_ {H} ^ {- 8} + \epsilon^ {4} \epsilon_ {H} ^ {- 1 1}\right) \tag {5}
$$
over all cases of  $\epsilon_H$ .
C Proof of Lemmas
Proof. According to the definition of  $v_{t}^{(i)}$ , we have
$$
\begin{array}{l} \frac {v _ {t + 1} ^ {(i)} - \nabla f _ {i} \left(x _ {t + 1} ^ {(i)}\right)}{(1 - \beta) ^ {t + 1}} - \frac {v _ {t} ^ {(i)} - \nabla f _ {i} \left(x _ {t} ^ {(i)}\right)}{(1 - \beta) ^ {t}} = \frac {\beta \left(\nabla F _ {i} \left(x _ {t + 1} ^ {(i)} , \xi_ {t + 1} ^ {(i)}\right) - \nabla f _ {i} \left(x _ {t + 1} ^ {(i)}\right)\right)}{(1 - \beta) ^ {t + 1}} \\ + \frac {\left(\nabla F _ {i} \left(x _ {t + 1} ^ {(i)} , \xi_ {t + 1} ^ {(i)}\right) - \nabla f _ {i} \left(x _ {t + 1} ^ {(i)}\right)\right) - \left(\nabla F _ {i} \left(x _ {t} ^ {(i)} , \xi_ {t + 1} ^ {(i)}\right) - \nabla f _ {i} \left(x _ {t} ^ {(i)}\right)\right)}{(1 - \beta) ^ {t}} \tag {6} \\ \end{array}
$$
where  $|\xi_{t + 1}^{(i)}| = b_1$ . The expectation of the right side of Eq. (6) over  $\xi_{t + 1}^{(i)}$  is 0. Using Cauchy-Schwartz inequality, Assumption 2 and Assumption 3 we have
$$
\begin{array}{l} \| \frac {\beta (\nabla F _ {i} (x _ {t + 1} ^ {(i)} , j) - \nabla f _ {i} (x _ {t + 1} ^ {(i)}))}{(1 - \beta) ^ {t + 1}} + \frac {(\nabla F _ {i} (x _ {t + 1} ^ {(i)} , j) - \nabla f _ {i} (x _ {t + 1} ^ {(i)})) - (\nabla F _ {i} (x _ {t} ^ {(i)} , j) - \nabla f _ {i} (x _ {t} ^ {(i)}))}{(1 - \beta) ^ {t}} \| ^ {2} \\ \leq \frac {2 \beta^ {2} \sigma^ {2}}{(1 - \beta) ^ {2 t + 2}} + \frac {8 L ^ {2} \| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \| ^ {2}}{(1 - \beta) ^ {2 t}} \tag {7} \\ \end{array}
$$
for each  $j \in \xi_{t+1}^{(i)}$ . Thus, applying Azuma-Hoeffding inequality to Eq. (6) we can obtain
$$
\begin{array}{l} \left\| v _ {t} ^ {(i)} - \nabla f _ {i} \left(x _ {t} ^ {(i)}\right) - (1 - \beta) ^ {t} \left(v _ {0} ^ {(i)} - \nabla f _ {i} \left(x _ {0}\right)\right) \right\| ^ {2} \\ \leq \frac {4 \log (4 / \delta)}{b _ {1}} \left(2 \beta \sigma^ {2} + 8 L ^ {2} \sum_ {s = 0} ^ {t - 1} (1 - \beta) ^ {2 (t - s)} \| x _ {s + 1} ^ {(i)} - x _ {s} ^ {(i)} \| ^ {2}\right) \tag {8} \\ \end{array}
$$
with probability  $1 - \delta$ . Here we use the fact that  $\sum_{s=0}^{+\infty}(1 - \beta)^s = \frac{1}{\beta}$ . Using Cauchy-Schwartz inequality to Eq. (8) we have
$$
\begin{array}{l} \| v _ {t} ^ {(i)} - \nabla f _ {i} (x _ {t} ^ {(i)}) \| ^ {2} \leq \frac {1 6 \log (4 / \delta)}{b _ {1}} (\beta \sigma^ {2} + 4 L ^ {2} \sum_ {s = 0} ^ {t - 1} (1 - \beta) ^ {2 (t - s)} \| x _ {s + 1} ^ {(i)} - x _ {s} ^ {(i)} \| ^ {2}) \\ + 2 (1 - \beta) ^ {2 t} \| v _ {0} ^ {(i)} - \nabla f _ {i} \left(x _ {0}\right)) \| ^ {2} \tag {9} \\ \end{array}
$$
Sum Eq. (9), we obtain
$$
\begin{array}{l} \frac {1}{\log (4 / \delta) n T} \sum_ {t = 0} ^ {T - 1} \sum_ {i = 1} ^ {n} \| v _ {t} ^ {(i)} - \nabla f _ {i} \left(x _ {t} ^ {(i)}\right) \| ^ {2} \\ \leq \frac {1 6 \beta \sigma^ {2}}{b _ {1}} + \frac {6 4 L ^ {2}}{n b _ {1} \beta T} \sum_ {t = 0} ^ {T - 2} \| X _ {t + 1} - X _ {t} \| _ {F} ^ {2} + \frac {2 \sigma^ {2}}{\beta b _ {0} T} \\ \leq \frac {1 6 \beta \sigma^ {2}}{b _ {1}} + \frac {3 8 4 L ^ {2}}{n b _ {1} \beta T} \sum_ {t = 0} ^ {T - 1} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \frac {1 9 2 L ^ {2}}{b _ {1} \beta T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} + \frac {2 \sigma^ {2}}{\beta b _ {0} T} \tag {10} \\ \end{array}
$$
which finishes the proof of (a) in Lemma 1. In the first inequality of Eq. (10) we apply Azuma-Hoeffding inequality to  $v_0^{(i)} - \nabla f_i(x_0)$ . In the second inequality we apply Cauchy-Schwartz inequality and use the fact  $x_{t + 1}^{(i)} - x_t^{(i)} = (x_{t + 1}^{(i)} - \bar{x}_{t + 1}) - (x_t^{(i)} - \bar{x}_t) + (\bar{x}_{t + 1} - \bar{x}_t)$ . Mimic above steps and we can achieve the inequality (b) in Lemma 1. The term  $n$  in the denominator is derived by the fact that  $\xi_t^{(i)},$  s on different nodes are independent.
Proof. As  $Y_{t} = W(Y_{t - 1} + V_{t} - V_{t - 1})$ , we have
$$
\begin{array}{l} \| Y _ {t} - \bar {Y} _ {t} \| _ {F} ^ {2} \\ = \| (W - J) \left(Y _ {t - 1} - \bar {Y} _ {t - 1}\right) + (W - J) \left(V _ {t} - V _ {t - 1}\right) \| _ {F} ^ {2} \\ \leq \lambda^ {2} \| Y _ {t - 1} - \bar {Y} _ {t - 1} \| _ {F} ^ {2} + 2 \langle (W - J) Y _ {t}, (W - J) \left(V _ {t} - V _ {t - 1}\right) \rangle + \lambda^ {2} \| V _ {t} - V _ {t - 1} \| _ {F} ^ {2} \\ \leq \frac {1 + \lambda^ {2}}{2} \| Y _ {t - 1} - \bar {Y} _ {t - 1} \| _ {F} ^ {2} + \frac {\lambda^ {2} + \lambda^ {4}}{1 - \lambda^ {2}} \| V _ {t} - V _ {t - 1} \| _ {F} ^ {2} \\ \leq \frac {1 + \lambda^ {2}}{2} \| Y _ {t - 1} - \bar {Y} _ {t - 1} \| _ {F} ^ {2} + \frac {3 \lambda^ {2} (1 + \lambda^ {2})}{1 - \lambda^ {2}} \sum_ {i = 1} ^ {n} \left(\| v _ {t} ^ {(i)} - \nabla f _ {i} \left(x _ {t} ^ {(i)}\right) \| ^ {2} + \| v _ {t - 1} ^ {(i)} - \nabla f _ {i} \left(x _ {t - 1} ^ {(i)}\right) \| ^ {2}\right) \\ + \frac {9 L ^ {2} \lambda^ {2} \left(1 + \lambda^ {2}\right)}{1 - \lambda^ {2}} \left(\| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \| X _ {t - 1} - \bar {X} _ {t - 1} \| _ {F} ^ {2} + n \omega_ {t - 1}\right) \tag {11} \\ \end{array}
$$
where the first inequality is derived by Assumption 5, the second inequality is derived by Young's inequality and the last inequality is derived by Cauchy-Schwartz inequality and Assumption 3. When  $t = 0$ , by Azuma-Hoeffding inequality we can get
$$
\left\| Y _ {0} - \bar {Y} _ {0} \right\| _ {F} ^ {2} \leq 2 \lambda^ {2} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} \left(x _ {0}\right) \| ^ {2} + \frac {8 \log (4 / \delta) n \lambda^ {2} \sigma^ {2}}{b _ {0}} \tag {12}
$$
with probability  $1 - \delta$ . As  $X_{t + 1} = W(X_t + \Omega_t)$ , by Assumption 5 and Young's inequality we have
$$
\begin{array}{l} \left\| X _ {t + 1} - \bar {X} _ {t + 1} \right\| _ {F} ^ {2} \\ \leq \frac {1 + \lambda^ {2}}{2} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \frac {2 \lambda^ {2}}{1 - \lambda^ {2}} \| \Omega_ {t} - \bar {\Omega} _ {t} \| _ {F} ^ {2} \\ \leq \frac {1 + \lambda^ {2}}{2} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \frac {4 \eta^ {2} \lambda^ {2}}{1 - \lambda^ {2}} \| Y _ {t} - \bar {Y} _ {t} \| _ {F} ^ {2} + \frac {4 \lambda^ {2}}{1 - \lambda^ {2}} \| \Omega_ {t} - \bar {\Omega} _ {t} - \eta (Y _ {t} - \bar {Y} _ {t}) \| _ {F} ^ {2} \\ \leq \frac {1 + \lambda^ {2}}{2} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \frac {4 \eta^ {2} \lambda^ {2}}{1 - \lambda^ {2}} \| Y _ {t} - \bar {Y} _ {t} \| _ {F} ^ {2} + \frac {8 n \lambda^ {2} p _ {t} \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{1 - \lambda^ {2}} \tag {13} \\ \end{array}
$$
where the second inequality is obtained by Cauchy-Schwartz inequality and the last inequality is because when node  $i$  draws perturbation it must satisfy  $\| y_t^{(i)}\| \leq C_v$ . Note that  $X_0 = \bar{X}_0$ . Sum Eq. (13), we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\| X _ {t} - \bar {X} _ {t} \right\| _ {F} ^ {2} \\ \leq \frac {8 \eta^ {2} \lambda^ {2}}{(1 - \lambda^ {2}) ^ {2}} \sum_ {t = 0} ^ {T - 1} \| Y _ {t} - \bar {Y} _ {t} \| _ {F} ^ {2} + \frac {1 6 n \lambda^ {2} (\eta^ {2} C _ {v} ^ {2} + r ^ {2}) T}{(1 - \lambda^ {2}) ^ {2}} \\ \leq \frac {2 8 8 L ^ {2} \eta^ {2} \lambda^ {4} (1 + \lambda^ {2})}{(1 - \lambda^ {2}) ^ {4}} \sum_ {t = 0} ^ {T - 1} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \frac {9 6 \eta^ {2} \lambda^ {4} (1 + \lambda^ {2})}{(1 - \lambda^ {2}) ^ {4}} \sum_ {t = 0} ^ {T - 1} \sum_ {i = 1} ^ {n} \| v _ {t} ^ {(i)} - \nabla f _ {i} (x _ {t} ^ {(i)}) \| ^ {2} \\ + \frac {1 4 4 n L ^ {2} \eta^ {2} \lambda^ {4} (1 + \lambda^ {2})}{(1 - \lambda^ {2}) ^ {4}} \sum_ {t = 0} ^ {T - 1} \omega_ {t} + \frac {1 6 \lambda^ {2} \eta^ {2}}{(1 - \lambda^ {2}) ^ {3}} \| Y _ {0} - \bar {Y} _ {0} \| _ {F} ^ {2} + \sum_ {t = 0} ^ {T - 1} \frac {1 6 n \lambda^ {2} p _ {t} (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{(1 - \lambda^ {2}) ^ {2}} \tag {14} \\ \end{array}
$$
where the last inequality comes from Eq. (11). When  $\eta \leq \frac{(1 - \lambda)^2}{40\lambda^2L}$  we have  $\frac{288L^2\eta^2\lambda^4(1 + \lambda^2)}{(1 - \lambda^2)^4} \leq \frac{1}{2}$  and
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} \\ \leq \frac {1 9 2 \eta^ {2} \lambda^ {4} (1 + \lambda^ {2})}{(1 - \lambda^ {2}) ^ {4} T} \sum_ {t = 0} ^ {T - 1} \sum_ {i = 1} ^ {n} \| v _ {t} ^ {(i)} - \nabla f _ {i} \left(x _ {t} ^ {(i)}\right) \| ^ {2} + \frac {2 8 8 n L ^ {2} \eta^ {2} \lambda^ {4} (1 + \lambda^ {2})}{(1 - \lambda^ {2}) ^ {4} T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} \\ \end{array}
$$
$$
\begin{array}{l} + \frac {6 4 \lambda^ {4} \eta^ {2}}{(1 - \lambda^ {2}) ^ {3} T} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} + \frac {2 5 6 \log (4 / \delta) n \lambda^ {4} \eta^ {2} \sigma^ {2}}{(1 - \lambda^ {2}) ^ {3} b _ {0} T} + \sum_ {t = 0} ^ {T - 1} \frac {3 2 n \lambda^ {2} p _ {t} (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{(1 - \lambda^ {2}) ^ {2} T} \\ \leq \frac {7 3 7 2 8 \log (4 / \delta) L ^ {2} \eta^ {2} \lambda^ {4} (1 + \lambda^ {2})}{(1 - \lambda^ {2}) ^ {4} b _ {1} \beta T} \sum_ {t = 0} ^ {T - 1} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \frac {2 4 5 7 6 n \log (4 / \delta) L ^ {2} \eta^ {2} \lambda^ {4} (1 + \lambda^ {2})}{(1 - \lambda^ {2}) ^ {4} b _ {1} \beta T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} \\ + \frac {n \log (4 / \delta) \eta^ {2} \lambda^ {4} (1 + \lambda^ {2}) \sigma^ {2}}{(1 - \lambda^ {2}) ^ {4}} (\frac {3 0 7 2 \beta}{b _ {1}} + \frac {3 8 4}{\beta b _ {0} T}) + \frac {2 8 8 n L ^ {2} \eta^ {2} \lambda^ {4} (1 + \lambda^ {2})}{(1 - \lambda^ {2}) ^ {4} T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} \\ + \frac {6 4 \lambda^ {4} \eta^ {2}}{(1 - \lambda^ {2}) ^ {3} T} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} + \frac {2 5 6 \log (4 / \delta) n \lambda^ {4} \eta^ {2} \sigma^ {2}}{(1 - \lambda^ {2}) ^ {3} b _ {0} T} + \sum_ {t = 0} ^ {T - 1} \frac {3 2 n \lambda^ {2} p _ {t} (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{(1 - \lambda^ {2}) ^ {2} T} \tag {15} \\ \end{array}
$$
where the last inequality is achieved by Lemma 1. According to the parameter setting, we have
$$
\frac {7 3 7 2 8 \log (4 / \delta) L ^ {2} \eta^ {2} \lambda^ {4} (1 + \lambda^ {2})}{(1 - \lambda^ {2}) ^ {4} b _ {1} \beta} \leq \frac {1}{2}
$$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} \\ \leq \frac {1 6 0 0 0 0 n \log (4 / \delta) L ^ {2} \eta^ {2} \lambda^ {4}}{(1 - \lambda) ^ {4} \min  \{b _ {1} \beta , 1 \} T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} + \frac {1 2 2 8 8 n \log (4 / \delta) \beta \eta^ {2} \lambda^ {4} \sigma^ {2}}{(1 - \lambda) ^ {4} b _ {1}} + \frac {2 0 0 0 n \log (4 / \delta) \eta^ {2} \lambda^ {4} \sigma^ {2}}{(1 - \lambda) ^ {4} \beta b _ {0} T} \\ + \frac {1 2 8 \lambda^ {4} \eta^ {2}}{(1 - \lambda) ^ {3} T} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} + \sum_ {t = 0} ^ {T - 1} \frac {6 4 n \lambda^ {2} p _ {t} \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{(1 - \lambda) ^ {2} T} \tag {16} \\ \end{array}
$$
where we have used the condition  $\lambda \leq 1$  to simplify the inequality. Moreover, sum Eq. (11) and we can achieve