SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By  $L$ -smoothness and  $\mu$ -strongly convexity, we have for  $\forall u \in \mathbb{R}^d$  that
$$
f (u) \geq f (x ^ {k}) + \langle \nabla f (x ^ {k}), u - x ^ {k} \rangle + \frac {\mu}{2} \| u - x ^ {k} \| ^ {2},
$$
$$
f _ {i} (u) \geq f _ {i} \left(x ^ {k}\right) + \left\langle \nabla f _ {i} \left(x ^ {k}\right), u - x ^ {k} \right\rangle + \frac {1}{2 L} \left\| \nabla f _ {i} (u) - \nabla f _ {i} \left(x ^ {k}\right) \right\| ^ {2},
$$
thus we obtain for  $\forall u\in \mathbb{R}^d$
$$
\begin{array}{l} f (x ^ {k}) \leq f (u) - \langle \nabla f (x ^ {k}), u - x ^ {k} \rangle \\ - \max  \left\{\frac {\mu}{2} \| u - x ^ {k} \| ^ {2}, \frac {1}{2 L n} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (u) - \nabla f _ {i} \left(x ^ {k}\right) \| ^ {2} \right\}. \tag {22} \\ \end{array}
$$
Applying Young's inequality to (21) and using  $\eta_k \leq 1/(2L)$ , we reach
$$
\begin{array}{l} f \left(y ^ {k + 1}\right) \leq f \left(x ^ {k}\right) + \frac {\eta_ {k}}{2} \mathcal {G} ^ {k} - \frac {\eta_ {k}}{2} \left(1 - L \eta_ {k}\right) \| g ^ {k} \| ^ {2} \\ \leq f \left(x ^ {k}\right) + \frac {\eta_ {k}}{2} \mathcal {G} ^ {k} - \frac {\eta_ {k}}{4} \| g ^ {k} \| ^ {2}. \tag {23} \\ \end{array}
$$
Adding (17) in Lemma 5 to  $\left(\frac{2\gamma_k\beta}{\theta_{1,k}} + 2\gamma_k(1 - \beta)\right) \times (23) + 2\gamma_k \times (22)$  (where  $u = x^{\star}$ ) and  $\frac{2\gamma_k\beta\theta_2}{\theta_{1,k}} \times (22)$  (where  $u = w^k$ ) and  $\frac{2\gamma_k\beta(1 - \theta_{1,k} - \theta_2)}{\theta_{1,k}} \times (22)$  (where  $u = y^k$ ) and using the unbiasedness of  $g^k$ , we obtain
$$
\begin{array}{l} \frac {2 \gamma_ {k} \beta}{\theta_ {1 , k}} \mathbb {E} _ {k} \left[ \mathcal {Y} ^ {k + 1} \right] + \mathbb {E} _ {k} \left[ \mathcal {Z} ^ {k + 1} \right] \\ \leq \beta \mathcal {Z} ^ {k} + (1 - \beta - \mu \gamma_ {k}) \| x ^ {k} - x ^ {\star} \| ^ {2} + \left(\gamma_ {k} ^ {2} - \frac {\eta_ {k} \gamma_ {k} \beta}{2 \theta_ {1 , k}}\right) \mathbb {E} _ {k} [ \| g ^ {k} \| ^ {2} ] + \eta_ {k} \left(\frac {\gamma_ {k} \beta}{\theta_ {1 , k}} + \gamma_ {k} (1 - \beta)\right) \mathcal {G} ^ {k} \\ - \frac {\gamma_ {k} \beta \theta_ {2}}{L \theta_ {1 , k}} \mathcal {G} _ {w} ^ {k} - \frac {\gamma_ {k} \beta (1 - \theta_ {1 , k} - \theta_ {2})}{L \theta_ {1 , k}} \mathcal {G} _ {y} ^ {k} + \frac {2 \gamma_ {k} \beta \theta_ {2}}{\theta_ {1 , k}} \mathcal {W} ^ {k} + \frac {2 \gamma_ {k} \beta (1 - \theta_ {1 , k} - \theta_ {2})}{\theta_ {1 , k}} \mathcal {Y} ^ {k} \\ - 2 \gamma_ {k} (1 - \beta) \mathbb {E} _ {k} [ \mathcal {Y} ^ {k + 1} ] - \frac {\eta_ {k} \gamma_ {k} (1 - \beta)}{2} \mathbb {E} _ {k} [ \| g ^ {k} \| ^ {2} ] \\ \end{array}
$$
On top of that, by applying our choice of the parameters, it can be easily verified that  $1 - \beta -\mu \gamma_{k} = 0$ $\gamma_k^2 -\frac{\eta_k\gamma_k\beta}{2\theta_{1,k}} = 0,1 - \beta \leq \frac{\beta}{4\theta_{1,k}}$  which leads to (20).
Lemma 7 ([32], Lemma 3, 4, 5). Under Assumptions 1, 2, and 3, the iterates of Algorithm 1 satisfy the following inequalities:
$$
\mathbb {E} \left[ \mathcal {W} ^ {k + 1} \right] = (1 - p) \mathbb {E} \left[ \mathcal {W} ^ {k} \right] + p \mathbb {E} \left[ \mathcal {Y} ^ {k} \right], \tag {24}
$$
$$
\mathbb {E} \left[ \mathcal {G} ^ {k} \right] \leq \frac {2 \omega}{n} \mathbb {E} \left[ \mathcal {G} _ {w} ^ {k} \right] + \frac {2 \omega}{n} \mathbb {E} \left[ \mathcal {H} ^ {k} \right], \tag {25}
$$
$$
\mathbb {E} \left[ \mathcal {H} ^ {k + 1} \right] \leq \left(1 - \frac {\alpha}{2}\right) \mathbb {E} \left[ \mathcal {H} ^ {k} \right] + 2 p \left(1 + \frac {2 p}{\alpha}\right) \left(\mathbb {E} \left[ \mathcal {G} _ {w} ^ {k} \right] + \mathbb {E} \left[ \mathcal {G} _ {y} ^ {k} \right]\right). \tag {26}
$$
Now we define a Lyapunov function  $\Psi^k$  for  $k\geq 1$  as
$$
\Psi^ {k} = \lambda_ {k - 1} \mathcal {W} ^ {k} + \frac {2 \gamma_ {k - 1} \beta}{\theta_ {1 , k - 1}} \mathcal {Y} ^ {k} + \mathcal {Z} ^ {k} + \frac {1 0 \eta_ {k - 1} \omega (1 + \omega) \gamma_ {k - 1} \beta}{\theta_ {1 , k - 1} n} \mathcal {H} ^ {k}, \quad \forall k \geq 1, \tag {27}
$$
where  $\lambda_{k} = \frac{\gamma_{k}\beta}{p\theta_{1,k}} (\theta_{1,k} + \theta_{2} - p + \sqrt{(p - \theta_{1,k} - \theta_{2})^{2} + 4p\theta_{2}})$ . Furthermore, it is straightforward to verify that
$$
\frac {2 \gamma_ {k} \beta \theta_ {2}}{p \theta_ {1 , k}} \leq \lambda_ {k} \leq \frac {2 \gamma_ {k} \beta (\theta_ {1 , k} + \theta_ {2})}{p \theta_ {1 , k}}.
$$
Now we restate the convergence result in the strongly-convex case in Theorem 3 and prove it using Lemma 6, 7 and the Lyapunov function.
Theorem 4. If  $\mu >0$  and parameters satisfy  $\eta_{k}\equiv \eta = n\theta_{2} / (120\omega L)$ ,  $\theta_{1,k}\equiv \theta_1 = 1 / (3\sqrt{\kappa})$ ,  $\alpha = p = 1 / (1 + \omega)$ ,  $\gamma_{k}\equiv \gamma = \eta /(2\theta_{1} + \eta \mu)$ ,  $\beta = 2\theta_{1} / (2\theta_{1} + \eta \mu)$ , and  $\theta_{2} = 1 / (3\sqrt{n} +3n / \omega)$  then the number of communication rounds performed by ADIANA to find an  $\epsilon$ -accurate solution such that  $\mathbb{E}[f(\hat{x})] - \min_x f(x)\leq \epsilon$  is at most  $\mathcal{O}((\omega +(1 + \omega /\sqrt{n})\sqrt{\kappa})\ln (L\Delta /\epsilon))$
Proof. In the strongly-convex case, parameters  $\{\gamma_k\}_{k\geq 1}$  and  $\{\theta_{1,k}\}_{k\geq 1}$  are constants, then so is  $\lambda_{k}$ . Thus, we simply write  $\gamma \triangleq \gamma_{k}$ ,  $\theta_{1} \triangleq \theta_{1,k}$ , and  $\lambda \triangleq \lambda_{k}$  for all  $k \geq 1$ . Considering (20) +  $\lambda(24) + \frac{5\gamma\beta\eta}{4\theta_1}(25) + \frac{10\eta\omega(1+\omega)\gamma\beta}{n\theta_1}(26)$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \Psi^ {k + 1} \right] \leq \left(\frac {2 \gamma \beta \theta_ {2}}{\theta_ {1}} + (1 - p) \lambda\right) \mathcal {W} ^ {k} + \left(\frac {2 \gamma \beta \left(1 - \theta_ {1} - \theta_ {2}\right)}{\theta_ {1}} + p \lambda\right) \mathcal {Y} ^ {k} + \beta \mathcal {Z} ^ {k} \\ + \left(1 - \frac {1}{4 (1 + \omega)}\right) \frac {1 0 \eta \omega (1 + \omega) \gamma \beta}{\theta_ {1} n} \mathcal {H} ^ {k} - \left(\frac {\gamma \beta \theta_ {2}}{L \theta_ {1}} - \frac {1 2 5 \gamma \beta \eta \omega}{2 n \theta_ {1}}\right) \mathcal {G} _ {w} ^ {k} \\ - \left(\frac {\gamma \beta \left(1 - \theta_ {1} - \theta_ {2}\right)}{L \theta_ {1}} - \frac {6 0 \eta \omega \gamma \beta}{n \theta_ {1}}\right) \mathcal {G} _ {y} ^ {k}. \tag {28} \\ \end{array}
$$
By the definition of  $\lambda$ , we have
$$
\begin{array}{l} \frac {2 \gamma \beta \theta_ {2}}{\theta_ {1}} + (1 - p) \lambda = \lambda \left(1 - p + \frac {2 p \theta_ {2}}{\sqrt {(p - \theta_ {1} - \theta_ {2}) ^ {2} + 4 p \theta_ {2}} + \theta_ {1} + \theta_ {2} - p}\right) \\ = \lambda \left(1 - p + \frac {2 p \theta_ {2}}{2 \theta_ {2} + \frac {4 \theta_ {1} \theta_ {2}}{\sqrt {(p - \theta_ {1} - \theta_ {2}) ^ {2} + 4 p \theta_ {2}} - \theta_ {1} + \theta_ {2} + p}}\right) \\ \leq \lambda \left(1 - p + \frac {p}{1 + \frac {2 \theta_ {1}}{\left(p + \theta_ {1} + \theta_ {2}\right) - \theta_ {1} + \theta_ {2} + p}}\right) = \left(1 - \frac {p \theta_ {1}}{p + \theta_ {1} + \theta_ {2}}\right) \lambda , \tag {29} \\ \end{array}
$$
$$
\begin{array}{l} \frac {2 \gamma \beta (1 - \theta_ {1} - \theta_ {2})}{\theta_ {1}} + p \lambda = \frac {2 \gamma \beta}{\theta_ {1}} \left[ 1 - \theta_ {1} - \theta_ {2} + \frac {1}{2} \left(\theta_ {1} + \theta_ {2} - p + \sqrt {(p - \theta_ {1} - \theta_ {2}) ^ {2} + 4 p \theta_ {2}}\right) \right] \\ = \frac {2 \gamma \beta}{\theta_ {1}} \left(1 - \frac {2 p \theta_ {1}}{p + \theta_ {1} + \theta_ {2} + \sqrt {(p - \theta_ {1} - \theta_ {2}) ^ {2} + 4 p \theta_ {2}}}\right) \\ \leq \left(1 - \frac {p \theta_ {1}}{p + \theta_ {1} + \theta_ {2}}\right) \frac {2 \gamma \beta}{\theta_ {1}}. \tag {30} \\ \end{array}
$$
From the choice of  $\eta$ , it is easy to verify that
$$
\frac {\gamma \beta \theta_ {2}}{L \theta_ {1}} - \frac {5 \gamma \beta \eta \omega}{2 n \theta_ {1}} - \frac {6 0 \eta \omega \gamma \beta}{n \theta_ {1}} \geq 0, \tag {31}
$$
and further noting  $1 - \theta_{1} - \theta_{2}\geq \theta_{2}$
$$
\frac {\gamma \beta \left(1 - \theta_ {1} - \theta_ {2}\right)}{L \theta_ {1}} - \frac {6 0 \eta \omega \gamma \beta}{n \theta_ {1}} \geq 0. \tag {32}
$$
Plugging (29), (30), (31), and (32) into (28), we obtain
$$
\begin{array}{l} \mathbb {E} \left[ \Psi^ {k + 1} \right] \leq \left(1 - \min  \left\{\frac {p \theta_ {1}}{p + \theta_ {1} + \theta_ {2}}, \frac {\eta \mu}{2 \theta_ {1} + \eta \mu}, \frac {1}{4 (1 + \omega)} \right\}\right) \Psi^ {k} \\ \leq \left(1 - \frac {1}{\frac {p + \theta_ {1} + \theta_ {2}}{p \theta_ {1}} + \frac {2 \theta_ {1} + \eta \mu}{\eta \mu} + 4 (1 + \omega)}\right) \Psi^ {k} \\ \leq \left(1 - \frac {1}{2 5 0 \left(\omega + \left(1 + \frac {\omega}{\sqrt {n}}\right) \sqrt {\kappa}\right)}\right) \Psi^ {k}, \quad \forall k \geq 0, \tag {33} \\ \end{array}
$$
where  $\Psi^0 \coloneqq \lambda \mathcal{W}^0 + \frac{2\gamma\beta}{\theta_1}\mathcal{Y}^0 + \mathcal{Z}^0 + \frac{10\eta\omega(1 + \omega)\gamma\beta}{\theta_1n}\mathcal{H}^0$ . Note that since we use initialization  $y^0 = z^0 = w^0 = h_i^0 = h^0$ ,  $\forall 1 \leq i \leq n$ , we have  $\mathcal{W}^0 = \mathcal{Y}^0 \leq (L\Delta) / 2$ ,  $\mathcal{Z}^0 \leq \Delta$ ,  $\mathcal{H}^0 \leq L^2\Delta$ , which indicates that
$$
\Psi^ {0} \leq \frac {L}{2} \cdot (\lambda_ {W} + \lambda_ {Y} + \lambda_ {Z} + \lambda_ {H}) \Delta ,
$$
where  $\lambda_{W} = \lambda \geq \frac{2\gamma\beta\theta_{2}}{\theta_{1}p}$ ,  $\lambda_{Y} = \frac{2\gamma\beta}{\theta_{1}}$ ,  $\lambda_{Z} = \frac{2}{L}$ ,  $\lambda_{H} = \frac{20\eta\omega(1 + \omega)\gamma\beta L}{\theta_{1}n}$ . These coefficients have the following inequalities:
$$
\begin{array}{l} \lambda_ {W} + \lambda_ {Y} \geq \frac {4 \eta (\theta_ {2} + p)}{p (2 \theta_ {1} + \eta \mu) ^ {2}} = \frac {n \theta_ {2} (\theta_ {2} + p)}{3 0 \omega L p (2 / 3 \sqrt {\kappa} + n \theta_ {2} / 1 2 0 \omega \kappa) ^ {2}} \geq \frac {n \theta_ {2} (\theta_ {2} + p) \kappa}{1 5 \omega L p} \\ \geq \frac {\kappa}{1 3 5 L} \geq \frac {1}{2 7 0} \lambda_ {Z}, \\ \end{array}
$$
$$
\frac {3}{3 2} (\lambda_ {W} + \lambda_ {Y}) \geq \frac {\kappa}{1 4 4 0 L} \geq \frac {(1 + \omega) n \theta_ {2} ^ {2} \kappa}{1 6 0 \omega L} \geq \frac {4 0 \eta^ {2} \omega (1 + \omega) L}{(2 \theta_ {1} + \eta \mu) ^ {2} n} = \lambda_ {H}.
$$
Consequently, the initial value of the Lyapunov function can be bounded as
$$
\Psi^ {0} \leq 1 3 6 L (\lambda_ {W} + \lambda_ {Y}) \Delta ,
$$
which together with (33) further implies that
$$
\begin{array}{l} \min  \left\{\mathbb {E} \left[ f \left(w ^ {T}\right) \right], \mathbb {E} \left[ f \left(y ^ {T}\right) \right] \right\} - f ^ {\star} \\ \leq \min  \left\{\frac {1}{\lambda_ {W}}, \frac {1}{\lambda_ {Y}} \right\} \left(1 - \frac {1}{2 5 0 \left(\omega + \left(1 + \frac {\omega}{\sqrt {n}}\right) \sqrt {\kappa}\right)}\right) ^ {T} \Psi^ {0} \\ \leq 2 7 2 L \Delta \left(1 - \frac {1}{2 5 0 \left(\omega + \left(1 + \frac {\omega}{\sqrt {n}}\right) \sqrt {\kappa}\right)}\right) ^ {T}. \\ \end{array}
$$
Thus,  $\mathcal{O}\left(\left(\omega +\left(1 + \frac{\omega}{\sqrt{n}}\right)\sqrt{\kappa}\right)\ln \left(\frac{L\Delta}{\epsilon}\right)\right)$  iterations are sufficient to guarantee an  $\epsilon$ -solution.
D.2 Generally-convex case
In this subsection, we restate the convergence result in the generally-convex case as in Theorem 3 and prove it using Lemma 6, 7 and the Lyapunov function defined in (27).
Theorem 5. If  $\mu = 0$  and parameters satisfy  $\alpha = 1 / (1 + \omega)$ ,  $\beta = 1$ ,  $p = \theta_{2} = 1 / (3(1 + \omega))$ ,  $\theta_{1,k} = 9 / (k + 27(1 + \omega))$ ,  $\gamma_{k} = \eta_{k} / (2\theta_{1,k})$ , and
then the number of communication rounds performed by ADIANA to find an  $\epsilon$ -accurate solution such that  $\mathbb{E}[f(\hat{x})] - \min_x f(x) \leq \epsilon$  is provided by  $\mathcal{O}((1 + \omega / \sqrt{n})\sqrt{L\Delta / \epsilon} + (1 + \omega)\sqrt[3]{L\Delta / \epsilon})$ .
Proof. Considering (20) +  $\lambda_{k}(24) + \frac{5\gamma_{k}\beta\eta_{k}}{4\theta_{1,k}}(25) + \frac{10\eta_{k}\omega(1 + \omega)\gamma_{k}\beta}{n\theta_{1,k}}(26)$  and applying the choice of  $\theta_{2}, p$  and  $\alpha$ , we have
$$
\begin{array}{l} \mathbb {E} _ {k} \left[ \Psi^ {k + 1} \right] \\ \leq \left(\frac {2 \gamma_ {k} \beta \theta_ {2}}{\theta_ {1 , k}} + (1 - p) \lambda_ {k}\right) \mathcal {W} ^ {k} + \left(\frac {2 \gamma_ {k} \beta (1 - \theta_ {1 , k} - \theta_ {2})}{\theta_ {1 , k}} + p \lambda_ {k}\right) \mathcal {Y} ^ {k} + \beta \mathcal {Z} ^ {k} \\ + \left(1 - \frac {1}{4 (1 + \omega)}\right) \frac {1 0 \eta_ {k} \omega (1 + \omega) \gamma_ {k} \beta}{n \theta_ {1 , k}} \mathcal {H} ^ {k} - \left(\frac {\gamma_ {k} \beta \theta_ {2}}{L \theta_ {1 , k}} - \frac {5 \omega \gamma_ {k} \beta \eta_ {k}}{2 n \theta_ {1 , k}} - \frac {1 0 0 \eta_ {k} \omega \gamma_ {k} \beta}{9 n \theta_ {1 , k}}\right) \mathcal {G} _ {w} ^ {k} \\ - \left(\frac {\gamma_ {k} \beta \left(1 - \theta_ {1 , k} - \theta_ {2}\right)}{L \theta_ {1 , k}} - \frac {1 0 0 \eta_ {k} \omega \gamma_ {k} \beta}{9 n \theta_ {1 , k}}\right) \mathcal {G} _ {y} ^ {k}. \tag {34} \\ \end{array}
$$
Similar to the proof of Theorem 4, we can simplify (34) by validating
$$
\left\{ \begin{array}{l} \frac {2 \gamma_ {k} \beta \theta_ {2}}{\theta_ {1 , k}} + (1 - p) \lambda_ {k} \leq \left(1 - \frac {p \theta_ {1 , k}}{p + \theta_ {1 , k} + \theta_ {2}}\right) \lambda_ {k} \leq \left(1 - \frac {\theta_ {1 , k}}{3}\right) \lambda_ {k}, \\ \frac {2 \gamma_ {k} \beta (1 - \theta_ {1 , k} - \theta_ {2})}{\theta_ {1 , k}} + p \lambda_ {k} \leq \left(1 - \frac {p \theta_ {1 , k}}{p + \theta_ {1 , k} + \theta_ {2}}\right) \frac {2 \gamma_ {k} \beta}{\theta_ {1 , k}} \leq \left(1 - \frac {\theta_ {1 , k}}{3}\right) \frac {2 \gamma_ {k} \beta}{\theta_ {1 , k}}, \\ \frac {\gamma_ {k} \beta \theta_ {2}}{L \theta_ {1 , k}} - \frac {5 \omega \gamma_ {k} \beta \eta_ {k}}{2 n \theta_ {1 , k}} - \frac {1 0 0 \eta_ {k} \omega \gamma_ {k} \beta}{9 n \theta_ {1 , k}} \geq 0, \\ \frac {\gamma_ {k} \beta (1 - \theta_ {1 , k} - \theta_ {2})}{L \theta_ {1 , k}} - \frac {1 0 0 \eta_ {k} \omega \gamma_ {k} \beta}{9 n \theta_ {1 , k}} \geq 0, \end{array} \right.
$$
and then obtain
$$
\begin{array}{l} \mathbb {E} _ {k} \left[ \Psi^ {k + 1} \right] \leq \left(1 - \frac {\theta_ {1 , k}}{3}\right) \lambda_ {k} \mathcal {W} ^ {k} + \left(1 - \frac {\theta_ {1 , k}}{3}\right) \frac {2 \gamma_ {k}}{\theta_ {1 , k}} \mathcal {Y} ^ {k} + \mathcal {Z} ^ {k} \\ + \left(1 - \frac {1}{4 (1 + \omega)}\right) \frac {1 0 \eta_ {k} \omega (1 + \omega) \gamma_ {k}}{\theta_ {1 , k} n} \mathcal {H} ^ {k}. \tag {35} \\ \end{array}
$$
For  $\forall k\geq 1$  , we have  $\theta_{1,k}\leq \theta_{1,k - 1}$  and thus
$$
\begin{array}{l} \left(1 - \frac {\theta_ {1 , k}}{3}\right) \lambda_ {k} = \left(1 - \frac {3}{k + 2 7 (1 + \omega)}\right) \frac {\eta_ {k}}{2 p \theta_ {1 , k} ^ {2}} \left(\theta_ {1, k} + \sqrt {\theta_ {1 , k} ^ {2} + 4 p \theta_ {2}}\right) \\ \leq \left(1 - \frac {3}{k + 2 7 (1 + \omega)}\right) \frac {\eta_ {k}}{2 p \theta_ {1 , k} ^ {2}} \left(\theta_ {1, k - 1} + \sqrt {\theta_ {1 , k - 1} ^ {2} + 4 p \theta_ {2}}\right) \\ = \left(1 - \frac {3}{k + 2 7 (1 + \omega)}\right) \left(\frac {k + 2 7 (1 + \omega)}{k - 1 + 2 7 (1 + \omega)}\right) ^ {2} \frac {\eta_ {k}}{\eta_ {k - 1}} \lambda_ {k - 1}. \\ \end{array}
$$
Further noting  $\frac{\eta_k}{\eta_{k - 1}}\leq 1 + \frac{1}{k + 27(1 + \omega)}$  , we obtain
$$
\begin{array}{l} \left(1 - \frac {\theta_ {1 , k}}{3}\right) \lambda_ {k} \leq \left(1 - \frac {3}{k + 2 7 (1 + \omega)}\right) \left(1 - \frac {1}{k + 2 7 (1 + \omega)}\right) ^ {- 2} \left(1 + \frac {1}{k + 2 7 (1 + \omega)}\right) \lambda_ {k - 1} \\ \leq \lambda_ {k - 1}. \tag {36} \\ \end{array}
$$
$$
\begin{array}{l} \left(1 - \frac {\theta_ {1 , k}}{3}\right) \frac {2 \gamma_ {k}}{\theta_ {1 , k}} \\ = \left(1 - \frac {3}{k + 2 7 (1 + \omega)}\right) \left(\frac {k + 2 7 (1 + \omega)}{k - 1 + 2 7 (1 + \omega)}\right) ^ {2} \frac {\eta_ {k}}{\eta_ {k - 1}} \frac {2 \gamma_ {k - 1}}{\theta_ {1 , k - 1}} \\ \leq \left(1 - \frac {3}{k + 2 7 (1 + \omega)}\right) \left(1 - \frac {1}{k + 2 7 (1 + \omega)}\right) ^ {- 2} \left(1 + \frac {1}{k + 2 7 (1 + \omega)}\right) \frac {2 \gamma_ {k - 1}}{\theta_ {1 , k - 1}} \\ \leq \frac {2 \gamma_ {k - 1}}{\theta_ {1 , k - 1}}, \tag {37} \\ \end{array}
$$
$$
\begin{array}{l} \left(1 - \frac {1}{4 (1 + \omega)}\right) \frac {1 0 \eta_ {k} \omega (1 + \omega) \gamma_ {k} \beta}{n \theta_ {1 , k}} \\ = \left(1 - \frac {1}{4 (1 + \omega)}\right) \left(\frac {k + 2 7 (1 + \omega)}{k - 1 + 2 7 (1 + \omega)}\right) ^ {2} \left(\frac {\eta_ {k}}{\eta_ {k - 1}}\right) ^ {2} \frac {1 0 \eta_ {k - 1} \omega (1 + \omega) \gamma_ {k - 1} \beta}{n \theta_ {1 , k - 1}} \\ \leq \frac {\left(1 - \frac {5}{k + 2 7 (1 + \omega)}\right) \left(1 + \frac {3}{k + 2 7 (1 + \omega)}\right)}{\left(1 - \frac {1}{k + 2 7 (1 + \omega)}\right) ^ {2}} \frac {1 0 \eta_ {k - 1} \omega (1 + \omega) \gamma_ {k - 1} \beta}{n \theta_ {1 , k - 1}} \\ \leq \frac {1 0 \eta_ {k - 1} \omega (1 + \omega) \gamma_ {k - 1} \beta}{\theta_ {1 , k - 1} n}. \tag {38} \\ \end{array}
$$
Combining (35),(36),(37), and (38), we have for  $\forall k\geq 1$  that
$$
\mathbb {E} _ {k} \left[ \Psi^ {k + 1} \right] \leq \Psi^ {k}. \tag {39}
$$
By applying (39) with  $k = T - 1, T - 2, \dots, 1$  and (35) with  $k = 0$ , we obtain
$$
\begin{array}{l} \mathbb {E} [ \Psi^ {T} ] \leq \left(1 - \frac {\theta_ {1 , 0}}{3}\right) \frac {2 \gamma_ {0} (\theta_ {1 , 0} + \theta_ {2})}{p \theta_ {1 , 0}} \mathcal {W} ^ {0} + \left(1 - \frac {\theta_ {1 , 0}}{3}\right) \frac {2 \gamma_ {0}}{\theta_ {1 , 0}} \mathcal {Y} ^ {0} + \mathcal {Z} ^ {0} \\ + \left(1 - \frac {1}{4 (1 + \omega)}\right) \frac {1 0 \eta_ {0} \omega (1 + \omega) \gamma_ {0}}{\theta_ {1 , 0} n} \mathcal {H} ^ {0} \\ \leq \frac {2}{L} \mathcal {W} ^ {0} + \frac {1}{L} \mathcal {Y} ^ {0} + \mathcal {Z} ^ {0} + \frac {3}{4 0 L ^ {2}} \mathcal {H} ^ {0} \leq \left(1 + \frac {1}{2} + 1 + \frac {3}{4 0}\right) \Delta \leq 3 \Delta . \\ \end{array}
$$
$$
\Psi^ {T} \geq \lambda_ {T - 1} \mathcal {W} ^ {T} + \frac {2 \gamma_ {T - 1} \beta}{\theta_ {1 , T - 1}} \mathcal {Y} ^ {T} \geq \frac {2 \gamma_ {T - 1} \beta \theta_ {2}}{\theta_ {1 , T - 1} p} \mathcal {W} ^ {T} + \frac {2 \gamma_ {T - 1} \beta}{\theta_ {1 , T - 1}} \mathcal {Y} ^ {T} = \frac {\eta_ {T - 1}}{\theta_ {1 , T - 1} ^ {2}} (\mathcal {W} ^ {T} + \mathcal {Y} ^ {T}),
$$
$$
\begin{array}{l} \max  \left\{\mathbb {E} \left[ f \left(w ^ {T}\right) \right], \mathbb {E} \left[ f \left(y ^ {T}\right) \right] \right\} - f ^ {\star} \\ \leq \frac {\theta_ {1 , T - 1} ^ {2}}{\eta_ {T - 1}} \mathbb {E} [ \Psi^ {T} ] \\ \leq \frac {2 4 3 \Delta}{(T - 1 + 2 7 (1 + \omega)) ^ {2}} \cdot \max  \left\{\frac {9 (1 + \omega) ^ {2} (1 + 2 7 (1 + \omega)) L}{T + 2 7 (1 + \omega)}, \frac {2 0 0 \omega (1 + \omega) L}{3 n}, 2 L \right\} \\ = \mathcal {O} \left(\frac {(1 + \omega^ {2} / n) L \Delta}{T ^ {2}} + \frac {(1 + \omega^ {3}) L \Delta}{T ^ {3}}\right), \\ \end{array}
$$
thus it suffices to achieve an  $\epsilon$ -solution with  $\mathcal{O}\left(\left(1 + \frac{\omega}{\sqrt{n}}\right)\sqrt{\frac{L\Delta}{\epsilon}} + (1 + \omega)\sqrt[3]{\frac{L\Delta}{\epsilon}}\right)$  iterations.
E Correction on CANITA [33]
We observe that when  $\omega \gg n$ , the original convergence rate of CANITA [33] contradicts the lower bounds presented in our Theorem 2. This discrepancy may stem from errors in the derivation of equations (35) and (36) in [33], or from the omission of certain conditions such as  $\omega = \Omega(n)$ . To address this issue, we provide a corrected proof and the corresponding convergence rate. Here we modify the choice of  $\beta_0$  in ([33], Theorem 2) to  $9(1 + b + \omega)^2 / (2(1 + b))$ , while keeping all other choices consistent with the original proof, i.e.,  $b = \min\{\omega, \sqrt{\omega(1 + \omega)^2 / n}\}$ ,  $p_t \equiv 1 / (1 + b)$ ,  $\alpha_t \equiv 1 / (1 + \omega)$ ,  $\theta_t = 3(1 + b) / (t + 9(1 + b + \omega))$ ,  $\beta = 48\omega(1 + \omega)(1 + b + 2(1 + \omega)) / (n(1 + b)^2)$  and
$$
\eta_ {t} = \left\{ \begin{array}{l l} \frac {1}{L (\beta_ {0} + 3 / 2)}, & \text {f o r} t = 0, \\ \min  \left\{\left(1 + \frac {1}{t + 9 (1 + b + \omega)}\right) \eta_ {t - 1}, \frac {1}{L (\beta + 3 / 2)} \right\}, & \text {f o r} t \geq 1. \end{array} \right.
$$
By definition we have
$$
\begin{array}{l} \eta_ {T} = \min \left\{\frac {T + 1 + 9 (1 + b + \omega)}{1 + 9 (1 + b + \omega)} \eta_ {0}, \frac {1}{L (\beta + 3 / 2)} \right\} \\ = \min  \left\{\frac {T + 1 + 9 (1 + b + \omega)}{1 + 9 (1 + b + \omega)} \frac {1}{L \left(\beta_ {0} + 3 / 2\right)}, \frac {1}{L (\beta + 3 / 2)} \right\} \\ \geq \min  \left\{\frac {(T + 9 (1 + b + \omega)) (1 + b)}{6 0 L (1 + b + \omega) ^ {3}}, \frac {1}{L (\beta + 3 / 2)} \right\} \tag {40} \\ \end{array}
$$
Plugging (40) and ([33],34) into ([33],33), we obtain
$$
\begin{array}{l} \mathbb {E} \left[ F ^ {T + 1} \right] = \mathcal {O} \left(\frac {(1 + b + \omega) ^ {3} L \Delta}{(T + 9 (1 + b + \omega)) ^ {3}} + \frac {(1 + b) (\beta + 3 / 2) L \Delta}{(T + 9 (1 + b + \omega)) ^ {2}}\right) \\ = \mathcal {O} \left(\frac {(1 + b + \omega) ^ {3} L \Delta}{T ^ {3}} + \frac {(1 + b) (\beta + 3 / 2) L \Delta}{T ^ {2}}\right). \tag {41} \\ \end{array}
$$
Using  $b = \min \{\omega, \sqrt{\omega(1 + \omega)^2 / n}\}$ , we have
$$
(1 + b + \omega) ^ {3} = \Theta \left((1 + \omega) ^ {3}\right),
$$
$$
\begin{array}{l} (1 + b) (\beta + 3 / 2) = \Theta \left((1 + b) + \frac {\omega (1 + \omega) (1 + b + \omega)}{n (1 + b)}\right) \\ = \Theta \left(1 + \frac {\omega^ {3 / 2}}{n ^ {1 / 2}} + \frac {\omega^ {2}}{n}\right), \\ \end{array}
$$
thus (41) can be simplified as
$$
\mathbb {E} \left[ F ^ {T + 1} \right] = \mathcal {O} \left(\frac {(1 + \omega) ^ {3} L \Delta}{T ^ {3}} + \frac {(1 + \omega^ {3 / 2} / n ^ {1 / 2} + \omega^ {2} / n) L \Delta}{T ^ {2}}\right).
$$
Consequently, for  $\epsilon < L\Delta / 2$  (i.e., a precision that the initial point does not satisfy), the communication rounds to achieve precision  $\epsilon$  is given by  $\mathcal{O}\left(\omega \frac{\sqrt[3]{L\Delta}}{\sqrt[3]{\epsilon}} + \left(1 + \frac{\omega^{3/4}}{n^{1/4}} + \frac{\omega}{\sqrt{n}}\right) \frac{\sqrt{L\Delta}}{\sqrt{\epsilon}}\right)$ .
F Experimental details and additional results
This section provides more details of the experiments listed in Sec. 6, as well as a few new experiments to validate our theories.
This section offers a comprehensive and detailed description of the experiments listed in Sec. 6, including problem formulation, data generation, cost calculation, and algorithm implementation.
Least squares. The local objective function of node  $i$  is defined as  $f_{i}(x) \coloneqq \frac{1}{2} \| A_{i}x - b_{i}\|^{2}$ , where  $A_{i} \in \mathbb{R}^{M \times d}$ ,  $b_{i} \in \mathbb{R}^{M}$ . We set  $d = 20$ ,  $M = 25$ , and the number of nodes  $n = 400$ . To generate  $A_{i}$ 's, we first randomly generate a Gaussian matrix  $G \in \mathbb{R}^{nM \times d}$ ; we then apply the SVD decomposition  $G = U\Sigma V^{\top}$  and replace the singular values in  $\Sigma$  by an arithmetic sequence starting from 1 and ending at 100 to get  $\tilde{\Sigma}$  and the resulted data matrix  $\tilde{G} = U\tilde{\Sigma} V^{\top}$ ; we finally allocate the submatrix of  $\tilde{G}$  composed of the  $((i - 1)M + 1)$ -th row to the  $(iM)$ -th row to be  $A_{i}$  for all  $1 \leq i \leq n$ .
Logistic regression. The local objective function of node  $i$  is defined as  $f_{i}(x) \coloneqq \frac{1}{M} \sum_{m=1}^{M} \ln(1 + \exp(-b_{i,m} a_{i,m}^{\top} x)$ , where number of nodes  $n = 400$ ,  $a_{i,m}$  stands for the feature of the  $m$ -th datapoint in the node  $i$ 's dataset, and  $b_{i,m}$  stands for the corresponding label. In a9a dataset, node  $i$  owns the  $(81(i-1)+1)$ -th to the  $(81i)$ -th datapoint with feature dimension  $d = 123$ . In w8a dataset, node  $i$  owns the  $(120(i-1)+1)$ -th to the  $(120i)$ -th datapoint with feature dimension  $d = 300$ .
Constructed problem. The local objective function of node  $i$  is defined as