SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| Y _ {t} - \bar {Y} _ {t} \| _ {F} ^ {2} \\ \leq \frac {1 2 \lambda^ {2}}{(1 - \lambda) T} \sum_ {t = 0} ^ {T - 1} \sum_ {i = 1} ^ {n} \| v _ {t} ^ {(i)} - \nabla f _ {i} \left(x _ {t} ^ {(i)}\right) \| ^ {2} + \frac {3 6 L ^ {2} \lambda^ {2}}{(1 - \lambda) T} \sum_ {t = 0} ^ {T - 1} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \frac {1 8 n L ^ {2} \lambda^ {2}}{(1 - \lambda) T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} \\ + \frac {2}{(1 - \lambda) T} \| Y _ {0} - \bar {Y} _ {0} \| _ {F} ^ {2} \\ \leq \frac {3 6 L ^ {2} \lambda^ {2}}{(1 - \lambda) T} \left(1 + \frac {1 2 8 \log (4 / \delta)}{b _ {1} \beta}\right) \sum_ {t = 0} ^ {T - 1} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \frac {1 8 n L ^ {2} \lambda^ {2}}{(1 - \lambda) T} \left(1 + \frac {1 2 8 \log (4 / \delta)}{b _ {1} \beta}\right) \sum_ {t = 0} ^ {T - 1} \omega_ {t} \\ + \frac {1 9 2 \log (4 / \delta) n \lambda^ {2} \beta \sigma^ {2}}{(1 - \lambda) b _ {1}} + \frac {2 5 \log (4 / \delta) n \lambda^ {2} \sigma^ {2}}{(1 - \lambda) \beta b _ {0} T} + \frac {4 \lambda^ {2}}{(1 - \lambda) T} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} \\ \leq \frac {4 6 4 4 \log (4 / \delta) L ^ {2} \lambda^ {2}}{(1 - \lambda) \min \{b _ {1} \beta , 1 \} T} \sum_ {t = 0} ^ {T - 1} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} + \frac {2 3 2 2 \log (4 / \delta) n L ^ {2} \lambda^ {2}}{(1 - \lambda) \min \{b _ {1} \beta , 1 \} T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} \\ + \frac {1 9 2 \log (4 / \delta) n \lambda^ {2} \beta \sigma^ {2}}{(1 - \lambda) b _ {1}} + \frac {2 5 \log (4 / \delta) n \lambda^ {2} \sigma^ {2}}{(1 - \lambda) \beta b _ {0} T} + \frac {4 \lambda^ {2}}{(1 - \lambda) T} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} \\ \leq \frac {3 7 1 5 2 \log (4 / \delta) L ^ {2} \eta^ {2} \lambda^ {4}}{(1 - \lambda) ^ {3} \min  \{b _ {1} \beta , 1 \} T} \sum_ {t = 0} ^ {T - 1} \| Y _ {t} - \bar {Y} _ {t} \| _ {F} ^ {2} + \frac {2 3 2 2 \log (4 / \delta) n L ^ {2} \lambda^ {2}}{(1 - \lambda) \min  \{b _ {1} \beta , 1 \} T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} \\ + \frac {1 9 2 \log (4 / \delta) n \lambda^ {2} \beta \sigma^ {2}}{(1 - \lambda) b _ {1}} + \sum_ {t = 0} ^ {T - 1} \frac {7 4 3 0 4 \log (4 / \delta) n L ^ {2} \lambda^ {4} p _ {t} (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{(1 - \lambda) ^ {3} \min  \{b _ {1} \beta , 1 \} T} + \frac {2 5 \log (4 / \delta) n \lambda^ {2} \sigma^ {2}}{(1 - \lambda) \beta b _ {0} T} \\ + \frac {4 \lambda^ {2}}{(1 - \lambda) T} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} \tag {17} \\ \end{array}
$$
where the second inequality uses Lemma 1 and Eq. (12). The last inequality uses the sum of Eq. (13). As  $\frac{37152\log(4 / \delta)L^2\eta^2\lambda^4}{(1 - \lambda)^3\min\{b_1\beta,1\}} \leq \frac{1}{2}$ , we have
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| Y _ {t} - \bar {Y} _ {t} \| _ {F} ^ {2} \\ \leq \frac {4 6 4 4 \log (4 / \delta) n L ^ {2} \lambda^ {2}}{(1 - \lambda) \min \{b _ {1} \beta , 1 \} T} \sum_ {t = 0} ^ {T - 1} \omega_ {t} + \frac {3 8 4 \log (4 / \delta) n \lambda^ {2} \beta \sigma^ {2}}{(1 - \lambda) b _ {1}} + \frac {5 0 \log (4 / \delta) n \lambda^ {2} \sigma^ {2}}{(1 - \lambda) \beta b _ {0} T} \\ + \frac {8 \lambda^ {2}}{(1 - \lambda) T} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} \left(x _ {0}\right) \| ^ {2} + \sum_ {t = 0} ^ {T - 1} \frac {1 5 0 0 0 0 \log (4 / \delta) n L ^ {2} \lambda^ {4} p _ {t} \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{(1 - \lambda) ^ {3} \min  \left\{b _ {1} \beta , 1 \right\} T} \tag {18} \\ \end{array}
$$
Proof. By Assumption 3 we have
$$
\begin{array}{l} f (\bar {x} _ {t + 1}) \leq f (\bar {x} _ {t}) + \langle \nabla f (\bar {x} _ {t}), \bar {x} _ {t + 1} - \bar {x} _ {t} \rangle + \frac {L}{2} \| \bar {x} _ {t + 1} - \bar {x} _ {t} \| ^ {2} \\ = f (\bar {x} _ {t}) + \left\langle \nabla f (\bar {x} _ {t}), - \eta \bar {v} _ {t} \right\rangle + \left\langle \nabla f (\bar {x} _ {t}), \bar {x} _ {t + 1} - \bar {x} _ {t} + \eta \bar {v} _ {t} \right\rangle + \frac {L}{2} \| \bar {x} _ {t + 1} - \bar {x} _ {t} \| ^ {2} \\ = f (\bar {x} _ {t}) - \frac {\eta}{2} \| \bar {v} _ {t} \| ^ {2} - \frac {\eta}{2} \| \nabla f (\bar {x} _ {t}) \| ^ {2} + \frac {\eta}{2} \| \bar {v} _ {t} - \nabla f (\bar {x} _ {t}) \| ^ {2} + \frac {\eta}{2} \| \nabla f (\bar {x} _ {t}) \| ^ {2} \\ + \frac {1}{2 \eta} \| \bar {x} _ {t + 1} - \bar {x} _ {t} + \eta \bar {v} _ {t} \| ^ {2} - \frac {1}{2 \eta} \| \bar {x} _ {t + 1} - \bar {x} _ {t} + \eta \bar {v} _ {t} - \eta \nabla f (\bar {x} _ {t}) \| ^ {2} + \frac {L}{2} \| \bar {x} _ {t + 1} - \bar {x} _ {t} \| ^ {2} \\ \leq f (\bar {x} _ {t}) - \frac {\eta}{2} \| \bar {v} _ {t} \| ^ {2} + \frac {\eta}{2} \| \bar {v} _ {t} - \nabla f (\bar {x} _ {t}) \| ^ {2} + \frac {1}{2 \eta} \| \bar {x} _ {t + 1} - \bar {x} _ {t} + \eta \bar {v} _ {t} \| ^ {2} + \frac {L \omega_ {t}}{2} \\ - \frac {1}{2 \eta} \omega_ {t} - \frac {\eta}{2} \| \bar {v} _ {t} - \nabla f (\bar {x} _ {t}) \| ^ {2} + \frac {1}{4 \eta} \omega_ {t} + \eta \| \bar {v} _ {t} - \nabla f (\bar {x} _ {t}) \| ^ {2} \\ \leq f (\bar {x} _ {t}) - \frac {1}{4 \eta} \omega_ {t} - \frac {\eta}{2} \| \bar {v} _ {t} \| ^ {2} + \frac {p _ {t} \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{\eta} + \frac {L \omega_ {t}}{2} + 2 \eta \| \bar {v} _ {t} - \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} (x _ {t} ^ {(i)}) \| ^ {2} \\ + \frac {L ^ {2} \eta}{n} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} \tag {19} \\ \end{array}
$$
where the first inequality is obtained by Young's inequality and the last inequality is obtained by Cauchy-Schwartz inequality, Assumption 3 and the fact that perturbation is only drawn when  $\| y_{t}^{(i)}\| \leq C_{v}$  and  $n_t$  nodes draw perturbation in iteration  $t$ . Sum Eq. (19) and apply Lemma 1, we have
$$
\begin{array}{l} f (\bar {x} _ {T}) \leq f (x _ {0}) - \frac {1}{4 \eta} \sum_ {t = 0} ^ {T - 1} \omega_ {t} - \frac {\eta}{2} \sum_ {t = 0} ^ {T - 1} \| \bar {v} _ {t} \| ^ {2} + (1 + \frac {7 6 8 \log (4 / \delta)}{n b _ {1} \beta}) \frac {L ^ {2} \eta}{n} \sum_ {t = 0} ^ {T - 1} \| X _ {t} - \bar {X} _ {t} \| _ {F} ^ {2} \\ + \sum_ {t = 0} ^ {T - 1} \frac {p _ {t} \left(\eta^ {2} C _ {v} ^ {2} + r ^ {2}\right)}{\eta} + \frac {3 2 \log (4 / \delta) \beta \eta T \sigma^ {2}}{n b _ {1}} + \left(1 + \frac {3 8 4 \log (4 / \delta) L \eta}{n b _ {1} \beta}\right) \sum_ {t = 0} ^ {T - 1} L \omega_ {t} \\ + \frac {4 \log (4 / \delta) \eta \sigma^ {2}}{n \beta b _ {0}} \tag {20} \\ \end{array}
$$
According to the update of gradient tracker, we have  $\bar{y}_t = \bar{v}_t$ . By Lemma 10 we have
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \| ^ {2} = \omega_ {t} + \frac {1}{n} \| \left(X _ {t + 1} - \bar {X} _ {t + 1}\right) - \left(X _ {t} - \bar {X} _ {t}\right) \| _ {F} ^ {2} (21) \\ \frac {1}{n} \sum_ {i = 1} ^ {n} \| y _ {t} ^ {(i)} \| ^ {2} = \| \bar {y} _ {t} \| ^ {2} + \frac {1}{n} \| Y _ {t} - \bar {Y} _ {t} \| _ {F} ^ {2} (22) \\ \end{array}
$$
Divide the term  $\| \bar{v}_t\| ^2$  in Eq. (20) into three portions and we get
In the second inequality we use Eq. (22). In the third inequality we use Eq. (21) and Cauchy-Schwartz inequality. In the last inequality we use Lemma 2 and the coefficients are
$$
A _ {1} = 1 + \frac {3 8 4 \log (4 / \delta) L \eta}{n b _ {1} \beta} + (1 + \frac {7 6 8 \log (4 / \delta)}{n b _ {1} \beta} + \frac {(1 - \lambda) ^ {2}}{1 2 8 L ^ {2} \eta^ {2}}) \frac {1 6 0 0 0 0 \log (4 / \delta) L ^ {3} \eta^ {3} \lambda^ {4}}{(1 - \lambda) ^ {4} \min  \left\{b _ {1} \beta , 1 \right\}} + \frac {7 7 4 \log (4 / \delta) L \eta \lambda^ {2}}{(1 - \lambda)}
$$
$$
A _ {2} = \frac {3 2 \log (4 / \delta)}{n} + \left(1 + \frac {7 6 8 \log (4 / \delta)}{n b _ {1} \beta} + \frac {(1 - \lambda) ^ {2}}{1 2 8 L ^ {2} \eta^ {2}}\right) \frac {1 2 2 8 8 \log (4 / \delta) L ^ {2} \eta^ {2} \lambda^ {4}}{(1 - \lambda) ^ {4}} + \frac {6 4 \log (4 / \delta) \lambda^ {2}}{(1 - \lambda)}
$$
$$
A _ {3} = \frac {4 \log (4 / \delta)}{n} + \left(1 + \frac {7 6 8 \log (4 / \delta)}{n b _ {1} \beta} + \frac {(1 - \lambda) ^ {2}}{1 2 8 L ^ {2} \eta^ {2}}\right) \frac {2 0 0 0 \log (4 / \delta) L ^ {2} \eta^ {2} \lambda^ {4}}{(1 - \lambda) ^ {4}} + \frac {1 0 \log (4 / \delta) \lambda^ {2}}{(1 - \lambda)}
$$
$$
A _ {4} = 1 + \left(1 + \frac {7 6 8 \log (4 / \delta)}{n b _ {1} \beta} + \frac {(1 - \lambda) ^ {2}}{1 2 8 L ^ {2} \eta^ {2}}\right) \frac {6 4 \lambda^ {2} L ^ {2} \eta^ {2}}{(1 - \lambda) ^ {2}} + \frac {2 5 0 0 0 \log (4 / \delta) L ^ {2} \eta^ {2} \lambda^ {4}}{(1 - \lambda) ^ {3}}
$$
$$
A _ {5} = \left(1 + \frac {7 6 8 \log (4 / \delta)}{n b _ {1} \beta} + \frac {(1 - \lambda) ^ {2}}{1 2 8 L ^ {2} \eta^ {2}}\right) \frac {1 2 8 \lambda^ {4} L ^ {2} \eta^ {2}}{(1 - \lambda) ^ {3}} + \frac {2 \lambda^ {2}}{1 - \lambda}
$$
According to the parameter setting, we have  $A_{1} \leq \frac{1}{16L\eta}$ ,  $A_{2} \leq \frac{200\log(4 / \delta)}{(1 - \lambda)^{2}}$ ,  $A_{3} \leq \frac{40\log(4 / \delta)}{(1 - \lambda)^{2}}$ ,  $A_{4} \leq \frac{7}{4}$  and  $A_{5} \leq \frac{5}{1 - \lambda}$ . Therefore, we have
$$
\begin{array}{l} f (\bar {x} _ {T}) \leq f (x _ {0}) + \frac {4 0 \log (4 / \delta) \eta \sigma^ {2}}{(1 - \lambda) ^ {2} \beta b _ {0}} + \frac {5 \eta}{(1 - \lambda) n} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {0}) \| ^ {2} - \sum_ {t = 0} ^ {T - 1} \mathcal {D} _ {t} \\ \leq f \left(x _ {0}\right) + \frac {\sigma^ {2}}{L} + \frac {1}{n L} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} \left(x _ {0}\right) \| ^ {2} - \sum_ {t = 0} ^ {T - 1} \mathcal {D} _ {t} \tag {24} \\ \end{array}
$$
$$
\mathcal {D} _ {t} = \frac {1}{1 6 \eta} \omega_ {t} + \frac {(1 - \lambda) ^ {2}}{2 5 6 n \eta} \sum_ {i = 1} ^ {n} \| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \| ^ {2} + \frac {\eta}{2 n} \sum_ {i = 1} ^ {n} \| y _ {t} ^ {(i)} \| ^ {2} - \frac {2 0 0 \eta \epsilon^ {2} \sigma^ {2}}{(1 - \lambda) ^ {2} C _ {1} ^ {2}} - \frac {7 p _ {t} (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{4 \eta} \tag {25}
$$
which reaches the conclusion.
Proof. For convenience, the iteration that draws perturbation is considered to be included in the escaping phase. If an iteration belongs to type-A, i.e.,  $p_t \geq \frac{1}{5}$ , then at least  $n/5$  worker nodes are in the escaping phase. If an iteration belongs to type-C, we have  $\frac{1}{n} \sum_{i=1}^{n} \| y_t^{(i)} \|^2 \leq \frac{4C_v^2}{5}$ . Therefore, there are at least  $\frac{n}{5}$  worker nodes satisfying  $\| y_t^{(i)} \| \leq C_v$ , which also indicates that at least  $\frac{n}{5}$  worker nodes are in the escaping phase. Then if iteration  $t$  is either type-A or type-C, there must be  $n/5$  worker nodes in the escaping phase. We denote the set of these  $n/5$  worker nodes as  $\mathcal{E}_t$ . Furthermore, if this iteration  $t$  is not one of the last  $C_T$  iterations before termination, then there must exist  $n/10$  worker nodes out of  $\mathcal{E}_t$  such that they have not met the condition  $esc^{(i)} \geq C_T$  and will break the escaping phase before meeting the condition because of the termination criterion in Algorithm 1. We use  $\mathcal{B}_t$  to denote these worker nodes.
For each  $i \in \mathcal{B}_t$ , we have an interval  $[a_t^{(i)}, b_t^{(i)}]$  such that  $t \in [a_t^{(i)}, b_t^{(i)}]$  and node  $i$  enters escaping phase in iteration  $a_t^{(i)}$  and breaks escaping phase in iteration  $b_t^{(i)}$ . Besides, we also have
$$
b _ {t} ^ {(i)} - a _ {t} ^ {(i)} \leq C _ {T} \quad \text {a n d} \quad \| x _ {b _ {t} ^ {(i)}} ^ {(i)} - x _ {a _ {t} ^ {(i)}} ^ {(i)} \| \geq C _ {d}
$$
Then by Cauchy-Schwartz inequality we have
$$
C _ {d} ^ {2} \leq \left\| x _ {b _ {t} ^ {(i)}} ^ {(i)} - x _ {a _ {t} ^ {(i)}} ^ {(i)} \right\| ^ {2} \leq C _ {T} \sum_ {t = a _ {t} ^ {(i)}} ^ {b _ {t} ^ {(i)}} \left\| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \right\| ^ {2} \tag {26}
$$
Let  $a_{t} = \min_{i}\{a_{t}^{(i)}\}$  and  $b_{t} = \max_{i}\{b_{t}^{(i)}\}$ . It is easy to check that  $b_{t} - a_{t} \leq 2C_{T}$ . Next, we will perform the refining step. If  $t < t'$  are two iterations that are either type-A or type-C and  $t' \in [a_{t}, b_{t}]$ , then we make  $a_{t'} = a_{t}$  and  $b_{t'} = b_{t}$ . Let  $\mathcal{I} = \cup_{t}[a_{t}, b_{t}]$  for all type-A and type-C iterations  $t$ . Then  $I$  can be written as disjoint union of
$$
\mathcal {I} = \mathcal {I} _ {1} \cup \mathcal {I} _ {2} \cup \dots \cup \mathcal {I} _ {k} \tag {27}
$$
because if  $a_{t} \leq a_{t'} \leq b_{t}$  then  $[a_{t}, b_{t}]$  and  $[a_{t'}, b_{t'}]$  can be merged into one interval. Now we can see for each iteration  $t$  that is either type-A or type-C and  $t$  is not one of the last  $C_T$  iterations, we have  $t \in \mathcal{I}$ . Next we will estimate the descent over  $\mathcal{I}$ . Without loss of generality, we consider an interval  $\mathcal{I}_j$ .  $\mathcal{I}_j$  can be expressed by union  $\mathcal{I}_1 \cup \dots \cup \mathcal{I}_l$  where  $\mathcal{I}_m = [a_{t_m}, b_{t_m}]$  for some  $t_m, m = 1, \dots, l$ . Because of the refining step, we have each  $t_m$  is only included in interval  $\mathcal{I}_m$  and the intersection of any three intervals in  $\mathcal{I}_1, \dots, \mathcal{I}_l$  is  $\emptyset$ . According to Eq. (26) we have
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {t \in \mathcal {I} _ {m}} \| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \| ^ {2} \geq \frac {C _ {d} ^ {2}}{1 0 C _ {T}} \tag {28}
$$
since  $|\mathcal{B}_t| \geq \frac{n}{10}$ . Next, we will consider the intersection of  $\mathcal{I}_m$  and  $\mathcal{I}_{m+1}$ . Notice that when estimating Eq. (28) we only add the terms  $\| x_{t+1}^{(i)} - x_t^{(i)} \|^2$  on nodes  $i \in \mathcal{B}_{t_m}$  and in the intervals  $[a_{t_m}^{(i)}, b_{t_m}^{(i)}]$ . Therefore, for any node  $i \notin \mathcal{B}_{t_m} \cap \mathcal{B}_{t_{m+1}}$ , the terms used to estimate Eq. (28) will not be added repeatedly. If  $i \in \mathcal{B}_{t_m} \cap \mathcal{B}_{t_{m+1}}$ , we have  $[a_{t_m}^{(i)}, b_{t_m}^{(i)}]$  and  $[a_{t_{m+1}}^{(i)}, b_{t_{m+1}}^{(i)}]$  are disjoint because  $t_{m+1} \in [a_{t_{m+1}}^{(i)}, b_{t_{m+1}}^{(i)}]$  but  $t_{m+1} \notin [a_{t_m}^{(i)}, b_{t_m}^{(i)}]$  and a node cannot draw perturbation before breaking the escaping phase. Hence we can sum Eq. (28) over  $m$  and achieve
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {t \in \mathcal {I} _ {j}} \| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \| ^ {2} \geq \frac {l C _ {d} ^ {2}}{1 0 C _ {T}} \tag {29}
$$
Since the length of each  $\mathcal{J}_m$  is not larger than  $2C_T$ , we have
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {t \in \mathcal {I} _ {j}} \| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \| ^ {2} \geq \frac {\left| \mathcal {I} _ {j} \right| C _ {d} ^ {2}}{2 0 C _ {T} ^ {2}} \text {a n d} \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {t \in \mathcal {I}} \| x _ {t + 1} ^ {(i)} - x _ {t} ^ {(i)} \| ^ {2} \geq \frac {\left| \mathcal {I} \right| C _ {d} ^ {2}}{2 0 C _ {T} ^ {2}} \tag {30}
$$
Combining Eq. (30) and Lemma 3, we can estimate the descent over  $\mathcal{I}$  by
$$
\sum_ {t \in \mathcal {I}} \mathcal {D} _ {t} \geq | \mathcal {I} | \left(\frac {(1 - \lambda) ^ {2} C _ {d} ^ {2}}{5 1 2 0 \eta C _ {T} ^ {2}} - \frac {2 0 0 \eta \epsilon^ {2} \sigma^ {2}}{(1 - \lambda) ^ {2} C _ {1} ^ {2}} - \frac {7 (\eta^ {2} C _ {v} ^ {2} + r ^ {2})}{4 \eta}\right) \geq | \mathcal {I} | \cdot \frac {(1 - \lambda) ^ {2} C _ {2} ^ {2} \eta \epsilon^ {2}}{1 0 0 0 0} \tag {31}
$$
according to the parameter setting.
Proof. According to Lemma 3 and the definition of type-B iteration, we have
$$
\mathcal {D} _ {t} \geq \frac {\eta C _ {v} ^ {2}}{2 0} - \frac {2 0 0 \eta \epsilon^ {2}}{(1 - \lambda) ^ {2} C _ {1} ^ {2}} - \frac {7 r ^ {2}}{2 0 \eta} \geq \frac {\eta C _ {v} ^ {2}}{4 0} - \frac {2 0 0 \eta \epsilon^ {2} \sigma^ {2}}{(1 - \lambda) ^ {2} C _ {1} ^ {2}} \geq \frac {(1 - \lambda) ^ {2} C _ {2} ^ {2} \eta \epsilon^ {2}}{8 0 0 0 0 0 0} \tag {32}
$$
for all type-B iteration  $t$  where we have used the parameter setting.
Proof. Suppose the conclusion is not true and we will find the conflict. Thus, we have the assumption that there are at least  $\frac{n}{10}$  worker nodes satisfying  $d_{i} \leq 2C_{d}$ . First, we define
$$
w _ {t} ^ {(i)} = x _ {t} ^ {(i)} - x _ {t} ^ {(i) ^ {\prime}}, w _ {t} = \bar {x} _ {t} - \bar {x} _ {t} ^ {\prime}, \mathcal {H} = \nabla^ {2} f (\bar {x} _ {s}), \mathcal {H} ^ {(i)} = \nabla^ {2} f _ {i} (\bar {x} _ {s}), \mathcal {H} _ {t} ^ {(i)} = \nabla^ {2} F _ {i} (x _ {s} ^ {(i)}, \xi_ {t} ^ {(i)})
$$
$$
\begin{array}{l} \zeta_ {t} = \frac {1}{n} \sum_ {i = 1} ^ {n} (\nabla F _ {i} (x _ {t} ^ {(i)}, \xi_ {t} ^ {(i)}) - \nabla F _ {i} (\bar {x} _ {t}, \xi_ {t} ^ {(i)})) - (\nabla F _ {i} (x _ {t} ^ {(i) ^ {\prime}}, \xi_ {t} ^ {(i)}) - \nabla F _ {i} (\bar {x} _ {t} ^ {\prime}, \xi_ {t} ^ {(i)})) \\ - (1 - \beta) (\nabla F _ {i} (x _ {t - 1} ^ {(i)}, \xi_ {t} ^ {(i)}) - \nabla F _ {i} (\bar {x} _ {t - 1}, \xi_ {t} ^ {(i)})) - (\nabla F _ {i} (x _ {t - 1} ^ {(i) ^ {\prime}}, \xi_ {t} ^ {(i)}) - \nabla F _ {i} (\bar {x} _ {t - 1} ^ {\prime}, \xi_ {t} ^ {(i)})) \\ \end{array}
$$
$$
\nu_ {t} = \bar {v} _ {t} - \nabla f (\bar {x} _ {t}) - (\bar {v} _ {t} ^ {\prime} - \nabla f (\bar {x} _ {t} ^ {\prime})) - \zeta_ {t}
$$
$$
\bar {\Delta} _ {t} = \int_ {0} ^ {1} \left(\nabla^ {2} f \left(\bar {x} _ {t} ^ {\prime} + \theta \left(\bar {x} _ {t} - \bar {x} _ {t} ^ {\prime}\right)\right) - \mathcal {H}\right) d \theta
$$
$$
\Delta_ {t} ^ {(i)} = \int_ {0} ^ {1} (\nabla^ {2} f _ {i} (\bar {x} _ {t} ^ {\prime} + \theta (\bar {x} _ {t} - \bar {x} _ {t} ^ {\prime})) - \mathcal {H} ^ {(i)}) d \theta
$$
$$
\begin{array}{l} w _ {t} = w _ {t - 1} - \eta (\bar {v} _ {t - 1} - \bar {v} _ {t - 1} ^ {\prime}) \\ = w _ {t - 1} - \eta (\nabla f (\bar {x} _ {t - 1}) - \nabla f (\bar {x} _ {t - 1} ^ {\prime}) + \bar {v} _ {t - 1} - \nabla f (\bar {x} _ {t - 1}) - \bar {v} _ {t - 1} ^ {\prime} + \nabla f (\bar {x} _ {t - 1} ^ {\prime})) \\ = w _ {t - 1} - \eta \Big [ (\bar {x} _ {t - 1} - \bar {x} _ {t - 1} ^ {\prime}) \int_ {0} ^ {1} \nabla^ {2} f (\bar {x} _ {t - 1} ^ {\prime} + \theta (\bar {x} _ {t - 1} - \bar {x} _ {t - 1} ^ {\prime})) d \theta + \nu_ {t - 1} + \zeta_ {t - 1} \Big ] \\ = (I - \eta \mathcal {H}) w _ {t - 1} - \eta \left(\bar {\Delta} _ {t - 1} w _ {t - 1} + \nu_ {t - 1} + \zeta_ {t - 1}\right) \tag {33} \\ \end{array}
$$
Here term  $\zeta_t$  is yield from consensus error and does not exist in centralized algorithms. Applying recursion to Eq. (33), we can obtain
$$
w _ {t} = \left(I - \eta \mathcal {H}\right) ^ {t - s - 1} w _ {s + 1} - \eta \sum_ {\tau = s + 1} ^ {t - 1} \left(I - \eta \mathcal {H}\right) ^ {t - \tau - 1} \left(\bar {\Delta} _ {\tau} w _ {\tau} + \nu_ {\tau} + \zeta_ {\tau}\right) \tag {34}
$$
Let  $q_{t} = \eta \sum_{\tau = s + 1}^{t - 1}(I - \eta \mathcal{H})^{t - \tau -1}(\bar{\Delta}_{\tau}w_{\tau} + \nu_{\tau} + \zeta_{\tau})$ . We will prove
$$
\left\| q _ {t} \right\| \leq \frac {1}{2} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \tag {35}
$$
$$
\frac {1}{2} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \leq \| w _ {t} \| \leq \frac {3}{2} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \tag {36}
$$
because  $\| (I - \eta \mathcal{H})^{t - s - 1}w_{s + 1}\| = (1 + \eta \gamma)^{t - s - 1}p_s r_0$  according to the definition of  $w_{s + 1}$ . We define  $\bar{d} = \max_{s\leq t\leq s + C_T}\{\| \bar{x}_t - \bar{x}_s\| ,\| \bar{x}_t' - \bar{x}_s\|\}$ . Since at least  $\frac{n}{10}$  nodes satisfy  $d_i\leq 2C_d$ ,  $C_d = \tilde{O} (\epsilon^{1 - \alpha})$  and the averaged consensus error is bounded by  $O(\epsilon^{2(1 + \theta)})$ , we have
$$
d _ {i} \leq 3 C _ {d} \quad \text {a n d} \quad \bar {d} \leq \frac {1}{n} \sum_ {i = 1} ^ {n} d _ {i} \leq 3 C _ {d} \tag {37}
$$
To achieve Eq. (35), it is sufficient to prove
$$
\eta \sum_ {\tau = s + 1} ^ {t - 1} (1 + \eta \gamma) ^ {t - \tau - 1} \| \bar {\Delta} _ {\tau} w _ {\tau} \| + \| v _ {\tau} \| + \| \zeta_ {\tau} \| \leq \frac {1}{2} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \tag {38}
$$
$$
\left\| \nu_ {t} \right\| \leq \sqrt {\frac {4 \log (4 / \delta)}{b _ {1}}} \cdot \frac {(1 + \eta \gamma) ^ {t - s - 1} L p _ {s} r _ {0}}{t - s} + \frac {1}{1 2 \eta C _ {T}} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \tag {39}
$$
$$
\left\| \zeta_ {t} \right\| \leq 8 \left(\frac {1 + \lambda^ {2}}{2}\right) ^ {\frac {t - s - 1}{2}} L _ {\sqrt {p _ {s}}} r _ {0} + \frac {L \eta (1 + \eta \gamma) ^ {t - s - 1} L p _ {s} r _ {0}}{\sqrt {b _ {1}} (t - s)} + \frac {1}{1 2 \eta C _ {T}} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \tag {40}
$$
which can be derived by induction. When  $t = s + 1$ , the left side of Eq. (38) is 0 and thus the inequality is satisfied. Suppose Eq. (38) holds for  $t \leq t_0$ . When  $t = t_0 + 1$ , we have
$$
\begin{array}{l} \eta \sum_ {\tau = s + 1} ^ {t - 1} (1 + \eta \gamma) ^ {t - \tau - 1} \| \bar {\Delta} _ {\tau} w _ {\tau} \| \\ \leq \frac {3}{2} \eta \rho \bar {d} \sum_ {\tau = s + 1} ^ {t - 1} (1 + \eta \gamma) ^ {t - s - 2} p _ {s} r _ {0} \leq 5 \eta \rho C _ {d} C _ {T} (1 + \eta \gamma) ^ {t - s - 2} p _ {s} r _ {0} \\ \leq \frac {1}{6} (1 + \eta \gamma) ^ {t - s - 1} p _ {s} r _ {0} \tag {41} \\ \end{array}
$$
where we use Assumption 4 and the case of  $t \leq t_0$  in the first two inequalities. We use the parameter setting of  $C_d$  in the last inequality. Next, we will estimate the terms related to  $\nu_t$ . By Azuma-Hoeffding inequality we know Eq. (39) is satisfied when  $t = s + 1$ . We define
$$
\epsilon_ {t, i} = \left(\nabla F _ {i} \left(\bar {x} _ {t + 1}, \xi_ {t + 1} ^ {(i)}\right) - \nabla f _ {i} \left(\bar {x} _ {t + 1}\right)\right) - (1 - \beta) \left(\nabla F _ {i} \left(\bar {x} _ {t}, \xi_ {t + 1}\right) - \nabla f _ {i} \left(\bar {x} _ {t}\right)\right)
$$
$$
\epsilon_ {t, i} ^ {\prime} = \left(\nabla F _ {i} \left(\bar {x} _ {t + 1} ^ {\prime}, \xi_ {t + 1} ^ {(i)}\right) - \nabla f _ {i} \left(\bar {x} _ {t + 1} ^ {\prime}\right)\right) - (1 - \beta) \left(\nabla F _ {i} \left(\bar {x} _ {t} ^ {\prime}, \xi_ {t + 1}\right) - \nabla f _ {i} \left(\bar {x} _ {t} ^ {\prime}\right)\right)
$$
Then according to the definition of  $\nu_{t}$  we have
$$
\nu_ {t + 1} = (1 - \beta) \nu_ {t} + \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\epsilon_ {t, i} - \epsilon_ {t, i} ^ {\prime}\right) = \frac {1}{n} \sum_ {\tau = s} ^ {t} (1 - \beta) ^ {t - \tau} \sum_ {i = 1} ^ {n} \left(\epsilon_ {\tau , i} - \epsilon_ {\tau , i} ^ {\prime}\right) \tag {42}
$$
$$
\tilde {\Delta} _ {t, 1} ^ {(i)} = \int_ {0} ^ {1} (\nabla^ {2} F _ {i} (\bar {x} _ {t} ^ {\prime} + \theta (\bar {x} _ {t} - \bar {x} _ {t} ^ {\prime}), \xi_ {t} ^ {(i)}) - \mathcal {H} _ {t} ^ {(i)}) d \theta
$$
$$
\tilde {\Delta} _ {t, 2} ^ {(i)} = \int_ {0} ^ {1} (\nabla^ {2} F _ {i} (\bar {x} _ {t - 1} ^ {\prime} + \theta (\bar {x} _ {t - 1} - \bar {x} _ {t - 1} ^ {\prime}), \xi_ {t} ^ {(i)}) - \mathcal {H} _ {t} ^ {(i)}) d \theta
$$
$$
\hat {\Delta} _ {t, 1} ^ {(i)} = \int_ {0} ^ {1} (\nabla^ {2} F _ {i} (x _ {t} ^ {(i) ^ {\prime}} + \theta (x _ {t} ^ {(i)} - x _ {t} ^ {(i) ^ {\prime}}), \xi_ {t} ^ {(i)}) - \mathcal {H} _ {t} ^ {(i)}) d \theta
$$
$$
\hat {\Delta} _ {t, 2} ^ {(i)} = \int_ {0} ^ {1} \left(\nabla^ {2} F _ {i} \left(x _ {t - 1} ^ {(i) ^ {\prime}} + \theta \left(x _ {t - 1} ^ {(i)} - x _ {t - 1} ^ {(i) ^ {\prime}}\right), \xi_ {t} ^ {(i)}\right) - \mathcal {H} _ {t} ^ {(i)}\right) d \theta \tag {43}
$$
$$
\begin{array}{l} \epsilon_ {t, i} - \epsilon_ {t, i} ^ {\prime} \\ = \mathcal {H} _ {t + 1} ^ {(i)} w _ {t + 1} + \tilde {\Delta} _ {t + 1, 1} ^ {(i)} w _ {t + 1} - \mathcal {H} ^ {(i)} w _ {t + 1} - \Delta_ {t + 1} ^ {(i)} w _ {t + 1} + (1 - \beta) \left(\mathcal {H} ^ {(i)} w _ {t} + \Delta_ {t} ^ {(i)} w _ {t}\right) \\ - (1 - \beta) (\mathcal {H} _ {t + 1} ^ {(i)} w _ {t} + \tilde {\Delta} _ {t + 1, 2} ^ {(i)} w _ {t}) \\ = \left(\mathcal {H} _ {t + 1} ^ {(i)} - \mathcal {H}\right) \left(w _ {t + 1} - (1 - \beta) w _ {t}\right) + \left(\tilde {\Delta} _ {t + 1, 1} ^ {(i)} - \Delta_ {t + 1} ^ {(i)}\right) w _ {t + 1} + (1 - \beta) \left(\Delta_ {t} ^ {(i)} - \tilde {\Delta} _ {t + 1, 2} ^ {(i)}\right) w _ {t} \tag {44} \\ \end{array}
$$
According to Assumption 3 and Assumption 4, we have