SlowGuess/NeurIPS_2023_OCR huggingface dataset:
case. Next, the algorithm enters into the loop iterations. In each iteration, the master only sends current  $\boldsymbol{x}_t$  to the chosen client  $i_t$ . The  $i_t$ -th client computes  $\nabla f_{i_t}(\boldsymbol{x}_t)$  and sends it to the master (the first client). Then the master solves (inexactly) the local problem (Line 5 in Algorithm 1) to get an inexact solution  $\boldsymbol{x}_{t+1}$ . The communication complexity is 2 in this case. Thus, the total communication complexity of SVRS $^{1\mathrm{ep}}$  is  $2(n-1) + 2T$ . Note that  $\mathbb{E}T = 1/p$  and generally  $p = 1/n$ . We obtain that one epoch communication complexity is  $4n - 2$  in expectation.
We would like to emphasize that our setup differs from that in [41, 46], where the authors assume the nodes can perform calculations and transmit vectors in parallel. We recognize the significance of both setups. However, there are situations where communication is more expensive than computation. For instance, in a business network or communication network the communication between any two nodes can result in charges and the risk of information leakage. To mitigate these costs, we should reduce the frequency of communication. Thus, we focus on the nonparallel setting.
3.1.2 Convergence Analysis of SVRS
Based on the one-epoch method SVRS<sup>1ep</sup>, we could introduce our non-accelerated algorithm SVRS, which starts from  $\boldsymbol{w}_0 \in \mathbb{R}^d$  and repeatedly performs the update<sup>7</sup>
$$
\boldsymbol {w} _ {k + 1} = \operatorname {S V R S} ^ {\operatorname {l e p}} (f, \boldsymbol {w} _ {k}, \theta , p), \forall k \geq 0.
$$
Now we derive the convergence rate of SVRS<sup>8</sup>. The main technique we apply is replacing the Euclidean distance with the Bergman divergence. Denote the reference function
$$
h (\boldsymbol {x}) := f _ {1} (\boldsymbol {x}) + \frac {1}{2 \theta} \| \boldsymbol {x} \| ^ {2} - f (\boldsymbol {x}). \tag {8}
$$
By Assumption 1 and 1) in Proposition 2.5, we see that  $f_{i}$ 's are  $\sqrt{n}\delta$ -SS. i.e.,  $[f_1 - f](\cdot)$  is  $(\sqrt{n}\delta)$ -smooth. Thus,  $h(\cdot)$  is  $\left(\frac{1}{\theta} - \sqrt{n}\delta\right)$ -strongly convex and  $\left(\frac{1}{\theta} + \sqrt{n}\delta\right)$ -smooth if  $\theta < \frac{1}{\sqrt{n}\delta}$ , that is,
$$
0 \leq \frac {1 - \sqrt {n} \theta \delta}{2 \theta} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} \stackrel {(2)} {\leq} D _ {h} (\boldsymbol {x}, \boldsymbol {y}) \stackrel {(3)} {\leq} \frac {1 + \sqrt {n} \theta \delta}{2 \theta} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2}. \tag {9}
$$
Hence, if  $\sqrt{n}\theta\delta = \Theta(1)$ ,  $h(\cdot)$  is nearly a rescaled Euclidean norm since its condition number related to  $\|\cdot\|$  is  $\frac{1 + \sqrt{n}\theta\delta}{1 - \sqrt{n}\theta\delta} = \Theta(1)$ . Next, we employ the properties of the Bregman divergence  $D_h(\cdot, \cdot)$  to build the one-epoch progress of SVRS<sup>1ep</sup> as shown below:
Lemma 3.1 Suppose Assumption 1 holds. Let  $\pmb{w}^{+} = \mathrm{SVRS}^{1\mathrm{ep}}(f,\pmb{w}_{0},\theta ,p)$  with  $\theta = 1 / (4\sqrt{n}\delta)$ , and the approximated solution  $\pmb{x}_{t + 1}$  satisfies
$$
\left\| \nabla A _ {\theta} ^ {t} \left(\boldsymbol {x} _ {t + 1}\right) \right\| ^ {2} \leq \frac {\mu}{2 0 \theta} \left\| \boldsymbol {x} _ {t} - \underset {\boldsymbol {x} \in \mathbb {R} ^ {d}} {\arg \min } A _ {\theta} ^ {t} (\boldsymbol {x}) \right\| ^ {2}, \forall t \geq 0. \tag {10}
$$
See Algorithm 3 in Appendix D for the details.
Similar results for the popular loopless version [34] can also be derived, see Appendix D.5 for the detail.
Algorithm 2 Accelerated SVRS (AccSVRS)
1: Input:  $z_0 = y_0 \in \mathbb{R}^d, p, \tau \in (0, 1), \alpha, \theta > 0, K \in \{1, 2, \ldots\}$   
2: for  $k = 0, 1, 2, \ldots, K - 1$  do  
3:  $x_{k + 1} = \tau z_k + (1 - \tau)y_k$   
4:  $y_{k + 1} = \mathrm{SVRS}^{1\mathrm{ep}}(f, x_{k + 1}, \theta, p)$   
5:  $\mathcal{G}_{k + 1} = p(\nabla [f_1 - f_{j_k}](x_{k + 1}) - \nabla [f_1 - f_{j_k}](y_{k + 1}) + \frac{1}{\theta}(\pmb{x}_{k + 1} - \pmb{y}_{k + 1}))$ ,  $j_k \sim \mathrm{Unif}([n])$   
6:  $z_{k + 1} = \arg \min_{z \in \mathbb{R}^d} \frac{1}{2\alpha} \|z - z_k\|^2 + \langle \mathcal{G}_{k + 1}, z \rangle + \frac{3\mu}{20} \|z - y_{k + 1}\|^2 = \frac{z_k + 0.3\mu\alpha y_{k + 1} - \alpha \mathcal{G}_{k + 1}}{1 + 0.3\mu\alpha}$   
7: end for  
8: Output:  $y_K$
Then for all  $\pmb{x} \in \mathbb{R}^d$  that is independent of the indices  $i_1, i_2, \ldots, i_T$  in SVRS $^{\text{1ep}}(f, \pmb{w}_0, \theta, p)$ , we have
$$
\mathbb {E} f \left(\boldsymbol {w} ^ {+}\right) - f (\boldsymbol {x}) \leq \mathbb {E} p \langle \boldsymbol {x} - \boldsymbol {w} _ {0}, \nabla h \left(\boldsymbol {w} ^ {+}\right) - \nabla h \left(\boldsymbol {w} _ {0}\right) \rangle - \left(p - \frac {2}{9 n}\right) D _ {h} \left(\boldsymbol {w} _ {0}, \boldsymbol {w} ^ {+}\right) - \frac {2 \mu \theta}{5} D _ {h} \left(\boldsymbol {x}, \boldsymbol {w} ^ {+}\right). \tag {11}
$$
Remark 3.2 We note that some papers [10, 11] assume the smoothness and convexity of component functions, and adopt local updates for solving the proximal step. However, we replace these assumptions with a proximal approximately solvable assumption (10), which could even cover some nonsmooth and non-convex but proximal trackable component functions. We regard our assumption as more essential since the local updates can be viewed as partially solving this proximal step.
The proof of Lemma 3.1 is left in Appendix D.1. From Lemma 3.1, we find a well-behaved proximal operator is sufficient to ensure favorable progress. Finally, we establish the convergence rate and communication complexity of the SVRS method, and the proof is deferred to Appendix D.2.
Theorem 3.3 Suppose Assumption 1 holds. If in SVRS<sup>1ep</sup>(Algorithm 1), the hyperparameters are set as  $\theta = 1 / (4\sqrt{n}\delta)$ ,  $p = 1 / n$ , and the approximate solution  $\mathbf{x}_{t + 1}$  in each proximal step satisfies Eq. (10). Then for any error  $\varepsilon > 0$ , when
$$
k \geq K _ {1} := \max  \left\{2, \frac {5 \delta}{\mu \sqrt {n}} \right\} \log \frac {3 \left(1 + \frac {\delta}{\mu \sqrt {n}}\right) [ f (\boldsymbol {w} _ {0}) - f (\boldsymbol {x} _ {*}) ]}{\varepsilon},
$$
i.e., after  $\tilde{\mathcal{O}}(n + \sqrt{n}\delta/\mu)$  communications in expectation, we obtain that  $\mathbb{E}f(\boldsymbol{w}_k) - f(\boldsymbol{x}_*) \leq \varepsilon$ .
Remark 3.4 Our results enjoy the following advantages over SVRP [33]: The convergence of SVRP ([33, Theorem 2]) only applied to  $\mathbb{E}\| \pmb {w}_k - \pmb {x}_*\|^2$ , which can also be derived by our results from strong convexity:  $f(\pmb {w}_k) - f(\pmb {x}_*)\geq \frac{\mu}{2}\| \pmb {w}_k - \pmb {x}_*\|^2$ . However, the reverse is not applicable since we do not assume the smoothness of  $f$ , or indeed the smoothness coefficient is very large. Moreover, for ill-conditioned problems (e.g.,  $\delta /\mu \gg \sqrt{n}$ ), our step size  $1 / (4\sqrt{n}\delta)$  is much larger than  $\mu /(2\delta^2)$  used in SVRP, and the convergence rate is also faster than SVRP:  $\tilde{\mathcal{O}}\left(n + \sqrt{n}\delta /\mu\right)$  vs.  $\tilde{\mathcal{O}}\left(n + \delta^2 /\mu^2\right)$ . Finally, we do not need the strong convexity assumption of component functions.
3.2 Acceleration Version: AccSVRS
Now we apply the classical interpolation technique motivated by Katyusha X [6] to establish accelerated SVRS (AccSVRS, Algorithm 2). The main difference between AccSVRS and Katyusha X is due to the different choices of distance spaces. Specifically, we adopt  $D_{h}(\cdot ,\cdot)$  instead of the Euclidean distance used in Katyusha X. Thus, the gradient mapping step (corresponding to Step 2 in [6, (4.1)]) should be built on the reference function  $h(\cdot)$  defined in Eq. (8), i.e.,  $\nabla h(\pmb{x}_{k + 1}) - \nabla h(\pmb{y}_{k + 1})$  instead of  $(\pmb{x}_{k + 1} - \pmb{y}_{k + 1}) / \theta$ . Moreover, noting that  $\nabla h(\cdot)$  could involve the heavy gradient computing part  $\nabla f(\cdot)$ , we further employ its stochastic version (Step 5 in Algorithm 2) to reduce the overall communication complexity.
Next, we delve into the convergence analysis. We first give the core lemma for AccSVRS, which is also motivated by the framework of Katyusha X [6]. The proof is deferred to Appendix D.3.
Lemma 3.5 Suppose Assumption 1 holds, and  $\theta = 1 / (4\sqrt{n}\delta)$ ,  $p = 1 / n$ ,  $\alpha \leq n\theta / (2\tau)$  in Algorithm 2, where  $\mathrm{SVRS}^{1\mathrm{ep}}(f, \boldsymbol{x}_{k+1}, \theta, p)$  satisfies Eq. (10) in each iteration. Then for all  $\boldsymbol{x} \in \mathbb{R}^d$  that is independent of the random indices  $i_1^{(k)}, i_2^{(k)}, \ldots, i_T^{(k)}$  in  $\mathrm{SVRS}^{1\mathrm{ep}}(f, \boldsymbol{x}_{k+1}, \theta, p)$ , we have that
$$
\mathbb {E} _ {k} \frac {\alpha}{\tau} [ f (\boldsymbol {y} _ {k + 1}) - f (\boldsymbol {x}) ] \leq \mathbb {E} _ {k} (1 - \tau) \cdot \frac {\alpha}{\tau} [ f (\boldsymbol {y} _ {k}) - f (\boldsymbol {x}) ] + \frac {\left\| \boldsymbol {x} - \boldsymbol {z} _ {k} \right\| ^ {2}}{2} - \frac {1 + 0 . 3 \mu \alpha}{2} \| \boldsymbol {x} - \boldsymbol {z} _ {k + 1} \| ^ {2}. \tag {12}
$$
Finally, we present the convergence rate and communication complexity of AccSVRS based on Lemma 3.5, and the proof is left in Appendix D.4.
Theorem 3.6 Suppose Assumption 1 holds. Consider AccSVRS with the following hyperparameters
$$
\theta = \frac {1}{4 \sqrt {n \delta}}, p = \frac {1}{n}, \tau = \frac {1}{4} \min \left\{1, \frac {n ^ {1 / 4}}{2} \sqrt {\frac {\mu}{\delta}} \right\}, \alpha = \frac {\sqrt {n}}{8 \delta \tau},
$$
and Eq. (10) is satisfied in each iteration of  $\mathrm{SVRS}^{\mathrm{1ep}}(f, \boldsymbol{x}_{k+1}, \theta, p)$ . Then for any  $\varepsilon > 0$ , when
$$
k \geq K _ {2} := \max  \left\{4, 8 n ^ {- 1 / 4} \sqrt {\delta / \mu} \right\} \log \frac {2 [ f (\boldsymbol {y} _ {0}) - f (\boldsymbol {x} _ {*}) ]}{\varepsilon},
$$
i.e., after  $\tilde{\mathcal{O}}\left(n + n^{3/4}\sqrt{\delta/\mu}\right)$  communications in expectation, we obtain that  $\mathbb{E}f(\pmb{y}_k) - f(\pmb{x}_*) \leq \varepsilon$ .
Remark 3.7 Although roughly the same as the communication complexity obtained by Catalyzed SVRP in [33, Theorem 3], our results have the following advantages.
Fewer assumptions. Except for the strong convexity of  $f$  and AveSS of  $f_{i}$ 's, we do not need to assume component strong convexity appearing in [33, Assumption 2].
Inexact proximal step. Khaled and Jin [33, Theorem 3] require exact evaluations of the proximal operator, though they mention that this is only for the convenience of analysis. Our framework allows approximated solutions in each proximal step, and the approximation criterion (10) is error-independent, i.e., irrelevant to the final error  $\varepsilon$ . Since the local proximal function is strongly convex, we could solve the problem in a few steps if additionally assuming the smoothness of  $f_{1}$ .
Smoothness-free bound. As shown in [33, Appendix G.1] or Appendix C, even if an exact proximal step is allowed, a dependence on the smoothness coefficient would be introduced in the total communication iterations of Catalyzed SVRP, though only in a log scale. Our directly accelerated method has no dependence on the smoothness coefficient.
3.3 Gradient Complexity under Smooth Assumption
Due to the importance of total computation in the machine learning and optimization community, we consider a more common setup by additionally assuming that  $f_{1}$  is  $L$ -smooth with  $L \geq \delta \geq \mu > 0$ , which together with Assumption 1 facilitates the quantification of Eq. (10). Then we can compute the total gradient complexity for AccSVRS as shown below. By Proposition 2.5 and our assumptions,  $A_{\theta}^{t}(\pmb{x})$  is  $\left(\frac{1}{\theta} - \sqrt{n}\delta\right)$ -strongly convex and  $\left(\frac{1}{\theta} + L\right)$ -smooth. Using accelerated methods starting from  $\pmb{x}_{t}$ , we can guarantee that Eq. (10) holds after  $T_{\mathrm{app}} = \tilde{\mathcal{O}}\left(\sqrt{\frac{1 + \theta L}{1 - \sqrt{n}\theta\delta}}\right) = \tilde{\mathcal{O}}\left(1 + n^{-1/4}\sqrt{L/\delta}\right)$  iterations with the choice of  $\theta$  in Theorem 3.6. Hence, the total gradient calls in expectation are
$$
\mathcal {O} (n T _ {\mathrm {a p p}} \cdot K _ {2}) = \tilde {\mathcal {O}} \left(n + n ^ {3 / 4} \left(\sqrt {\delta / \mu} + \sqrt {L / \delta}\right) + \sqrt {n L / \mu}\right).
$$
Since  $\delta \in [\mu, L]$ , we recover the optimal gradient complexity  $\tilde{\mathcal{O}}(n + n^{3/4}\sqrt{L / \mu})$  for the average smooth setting [63, Table 1] if neglecting log factors. Particularly, when  $\delta = \Theta(\sqrt{\mu L})$ , we even obtain the nearly optimal gradient complexity  $\tilde{\mathcal{O}}(n + \sqrt{nL / \mu})$  for the component smooth setting [25, 26, 56]. We leave the details in Appendix E. Although the gradient complexity is not the primary focus of our work, we have demonstrated that the gradient complexity bound of AccSVRS is nearly optimal for certain values of  $\delta$  in specific cases.
In this section, we establish the lower bound of the communication complexity, which nearly matches the upper bound of AccSVRS.
4.1 Definition of Algorithms
In this subsection, we specify the class of algorithms to which our lower bound can apply. We first introduce the Proximal Incremental First-order Oracle (PIFO) [56, 25], which is defined as  $h_{f_i}^{\mathrm{P}}(\boldsymbol{x},\gamma) = [f_i(\boldsymbol{x}),\nabla f_i(\boldsymbol{x}),\mathrm{prox}_{f_i}^\gamma (\boldsymbol{x})]$  with  $\gamma >0$ . Here the proximal operator is defined as  $\mathrm{prox}_{f_i}^\gamma (\boldsymbol {x})\coloneqq \arg \min_{\boldsymbol {u}}\{f_i(\boldsymbol {u}) + \frac{1}{2\gamma}\| \boldsymbol {x} - \boldsymbol {u}\| ^2\}$ . In addition to the local zero-order and first-order information of  $f_{i}$  at  $\pmb{x}$ , the PIFO  $h_{f_i}^{\mathrm{P}}(\boldsymbol {x},\gamma)$  also provides some global information through the proximal operator<sup>9</sup>. Then we assume the algorithm has access to the PIFO and the definition of algorithms is presented as follows.
Definition 4.1 Consider a randomized algorithm  $\mathcal{A}$  to solve problem (1). Suppose the number of communication rounds is  $T$ . At the initialization stage, the master node 1 communicates with all the others. In round  $t$  ( $0 \leq t \leq T - 1$ ), the algorithm samples a node  $i_t \sim \mathrm{Unif}([n])$ , and node 1 communicates with node  $i_t$ . Then the algorithm samples a Bernoulli random variable  $a_t$  with constant expectation  $c_0 / n$ . If  $a_t = 1$ , node 1 communicates with all the others. Define the information set  $\mathcal{I}_{t + 1}$  as the set of all the possible points  $\mathcal{A}$  can obtain after round  $t$ . The algorithm updates  $\mathcal{I}_{t + 1}$  based on the linear-span operation and PIFO, and finally outputs a certain point in  $\mathcal{I}_T$ .
At the initialization stage, the communication cost is  $2(n - 1)$ . In each communication round, the Bernoulli random variable  $a_{t}$  determines whether the master node communicates with all the others, i.e., whether to calculate the full gradient. Since  $\mathbb{E}a_{t} = c_{0} / n$ , the expected communication cost of each round is of the order  $\Theta(1)$ . Thus the total communication cost is of the order  $\Theta(n + T)$  and we can use  $T$  to measure the communication complexity. Moreover, one can check Algorithm 2 satisfies Definition 4.1. The formal definition and detailed analysis are deferred to Appendix F.1.
4.2 The Construction and Results
In this section, we construct a hard instance of problem (1) and then use it to establish the lower bound. Due to space limitations, we only present several key properties. The complete framework of construction is deferred to Appendix F.2.
Inspired by [25], we consider the class of matrices  $\pmb{B}(m,\zeta) = \left[ \begin{array}{ccc}1 & -1 & \\ \ddots & \ddots & \\ & 1 & -1\\ & & \zeta \end{array} \right] \in \mathbb{R}^{m\times m}$ . This class of matrices is widely used to establish lower bounds for minimax optimization problems [61, 49, 59], and  $\pmb{A}(m,\zeta)\coloneqq \pmb{B}(m,\zeta)^{\top}\pmb{B}(m,\zeta)$  is the well-known tridiagonal matrix in the analysis of lower bounds for convex optimization [47, 40, 15]. Denote the  $l$ -th row of  $\pmb{B}(m,\zeta)$  as  $\pmb{b}_l(m,\zeta)^\top$ . We partition the row vectors of  $\pmb{B}(m,\zeta)$  according to the index sets  $\mathcal{L}_i = \{l:1\leq l\leq m,l\equiv i - 1 (\mathrm{mod} (n - 1))\}$  for  $2\leq i\leq n$  and  $\mathcal{L}_1 = \varnothing^{10}$ . These sets are mutually exclusive and their union is  $[m]$ . Then we consider the following problem
$$
\min  _ {\boldsymbol {x} \in \mathbb {R} ^ {m}} r (\boldsymbol {x}; m, \zeta , c) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ r _ {i} (\boldsymbol {x}; m, \zeta , c) := \left\{ \begin{array}{l l} \frac {c}{2} \| \boldsymbol {x} \| ^ {2} - n \langle \boldsymbol {e} _ {1}, \boldsymbol {x} \rangle & \text {f o r} i = 1, \\ \frac {c}{2} \| \boldsymbol {x} \| ^ {2} + \frac {n}{2} \sum_ {l \in \mathcal {L} _ {i}} \| \boldsymbol {b} _ {l} (m, \zeta) ^ {\top} \boldsymbol {x} \| ^ {2} & \text {f o r} i \neq 1. \end{array} \right. \right] \tag {13}
$$
Here  $e_i \in \mathbb{R}^m$  denotes the unit vector with the  $i$ -th element equal to 1 and others equal to 0. Then one can check  $r(\pmb{x}; m, \zeta, c) = \frac{1}{2} \pmb{x}^\top \pmb{A}(m, \zeta) \pmb{x} + \frac{c}{2} \| \pmb{x} \|^2 - \langle e_1, \pmb{x} \rangle$ . Clearly,  $r$  is  $c$ -strongly convex. We can also determine the AveSS parameter as follows. The proof is deferred to Appendix F.3.
Proposition 4.2 Suppose that  $0 < \zeta \leq \sqrt{2}$ ,  $n \geq 3$  and  $m \geq 3$ . Then  $r_i$ 's satisfy  $\sqrt{8n + 4}$ -AveSS.
Define the subspaces  $\{\mathcal{F}_k\}_{k=0}^m$  as  $\mathcal{F}_0 = \{\mathbf{0}\}$  and  $\mathcal{F}_k = \operatorname{span}\{e_1, e_2, \ldots, e_k\}$  for  $1 \leq k \leq m$ . The next lemma is fundamental to our analysis. The proof is deferred to Appendix F.5.
Lemma 4.3 Suppose the algorithm  $\mathcal{A}$  satisfies Definition 4.1 and apply it to solve problem (13) with  $n\geq 3$  and  $m\geq 4$ . We have (i)  $\mathcal{I}_0 = \mathcal{F}_1$ . (ii) Suppose  $\mathcal{I}_t\subseteq \mathcal{F}_k$  ( $1\leq k\leq m - 3$ ). If  $i_{t}$  satisfies  $k\in \mathcal{L}_{i_t}$  or  $a_{t} = 1$ , then  $\mathcal{I}_{t + 1}\subseteq \mathcal{F}_{k + 3}$ ; otherwise,  $\mathcal{I}_{t + 1}\subseteq \mathcal{F}_k$ .
If we let  $\gamma \to \infty$ ,  $\operatorname{prox}_{f_i}^\gamma(\boldsymbol{x})$  converges to the exact minimizer of  $f_i$ , irrelevant to the choice of  $\boldsymbol{x}$ .
10 Such a way of partitioning is also inspired by [25] and similar to that in [36]. However, our setting is different from theirs.
Figure 1: Numerical experiments on synthetic data. The corresponding coefficients are shown in the title of each graph. We plot the function gap on a log scale versus the number of communication steps, where one exchange of vectors counts as a communication step.
Lemma 4.3 guarantees that in each round, only when a specific component is sampled or the full gradient is calculated, can we expand the information set by at most three dimensions. For problem (13), we could never obtain an approximate solution unless we expand the information set to the whole space (see Proposition F.6 in Appendix F.2), while Lemma 4.3 implies that the process of expanding is very slow. Then we can establish the following lower bound.
Theorem 4.4 For any  $n \geq 3$ ,  $\delta, \mu > 0$ , algorithm  $\mathcal{A}$  satisfying Definition 4.1 and sufficiently small  $\epsilon > 0$ , there exists a rescaled version of problem (13) such that (i) Assumption 1 holds; (ii) In order to find an  $\epsilon$ -suboptimal solution  $\hat{\pmb{x}}$  such that  $\mathbb{E}r(\hat{\pmb{x}}) - \min_{\pmb{x}} r(\pmb{x}) < \epsilon$  by  $\mathcal{A}$ , the communication complexity in expectation is  $\tilde{\Omega}(n + n^{3/4}\sqrt{\delta/\mu})$ .
This lower bound nearly matches the upper bound in Theorem 3.6 up to log factors, implying Algorithm 2 is nearly optimal in terms of communication complexity. The detailed statement and proof are deferred to Appendices F.2 and F.9.
To demonstrate the advantages of our algorithms, we conduct the same numerical experiments as those in [35, 33]. We focus on the linear ridge regression problem with  $\ell_2$  regularization, where the average loss  $f$  has the formulation:  $f(\pmb{x}) = \frac{1}{n}\sum_{i=1}^{n}\left[f_i(\pmb{x})\coloneqq \frac{1}{m}\sum_{j=1}^{m}\left(\pmb{z}_{i,j}^\top \pmb{x} - y_{i,j}\right)^2 + \frac{\mu}{2}\|\pmb{x}\|^2\right]$ . Here  $\pmb{z}_{i,j} \in \mathbb{R}^d$  and  $y_{i,j} \in \mathbb{R}, \forall i \in [n], j \in [m]$  serve as the feature and label respectively, and  $m$  can be viewed as data size in each local client. We consider a synthetic dataset generated by adding a small random noise matrix to the center matrix, ensuring a small  $\delta$ . To capture the differences in convergence rates between our methods and SVRP caused by different magnitudes of  $\mu$ , we vary  $\mu = 10^{-i}, i \in \{0,1,2\}$ . We compare our methods (SVRS and AccSVRS) against SVRG, KatyushaX, SVRP (Catalyzed SVRP is somehow hard to tune so we omit it), and Accelerated Extragradient (AccEG) using their theoretical step sizes, except that we scale the interpolation parameter  $\tau$  in KatyushaX and AccSVRS for producing practical performance (see Appendix G for detail). From Figure 1, we can observe that for a large  $\mu$ , SVRP outperforms existing algorithms due to its high-order dependence on  $\mu$ . However, when the problem becomes ill-conditioned with a small  $\mu$ , AccSVRS exhibits significant improvements compared to other algorithms.
In this paper, we have introduced two new algorithms, SVRS and its directly accelerated version AccSVRS, and established improved communication complexity bounds for distributed optimization under the similarity assumption. Our rates are entirely smoothness-free and only require strong convexity of the objective, average similarity, and proximal friendliness of components. Moreover, our methods also have nearly optimal gradient complexity (leaving out the log term) when applied to smooth components in specific cases. It would be interesting to remove additional log terms to achieve both optimal communication and local gradient calls as [35], as well as investigating the complexity under other similarity assumptions (such as SS instead of AveSS) in future research.
Lin, Han, and Zhang have been supported by the National Key Research and Development Project of China (No. 2022YFA1004002) and the National Natural Science Foundation of China (No. 12271011). Ye has been supported by the National Natural Science Foundation of China (No. 12101491).