SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dachao Lin
Yuze Han
Haishan Ye
We study finite-sum distributed optimization problems involving a master node and  $n - 1$  local nodes under the popular  $\delta$ -similarity and  $\mu$ -strong convexity conditions. We propose two new algorithms, SVRS and AccSVRS, motivated by previous works. The non-accelerated SVRS method combines the techniques of gradient sliding and variance reduction and achieves a better communication complexity of  $\tilde{\mathcal{O}}(n + \sqrt{n}\delta/\mu)$  compared to existing non-accelerated algorithms. Applying the framework proposed in Katyusha X [6], we also develop a directly accelerated version named AccSVRS with the  $\tilde{\mathcal{O}}(n + n^{3/4}\sqrt{\delta/\mu})$  communication complexity. In contrast to existing results, our complexity bounds are entirely smoothness-free and exhibit superiority in ill-conditioned cases. Furthermore, we establish a nearly matched lower bound to verify the tightness of our AccSVRS method.
We have witnessed the development of distributed optimization in recent years. Distributed optimization aims to cooperatively solve a learning task over a predefined social network by exchanging information exclusively with immediate neighbors. This class of problems has found extensive applications in various fields, including machine learning, healthcare, network information processing, telecommunications, manufacturing, natural language processing tasks, and multi-agent control [54, 30, 48, 45, 60, 8]. In this paper, we focus on the following classical finite-sum optimization problem in a centralized setting:
$$
\min  _ {\boldsymbol {x} \in \mathbb {R} ^ {d}} f (\boldsymbol {x}) := \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (\boldsymbol {x}), \tag {1}
$$
where each  $f_{i}$  is differentiable and corresponds to a client or node, and the target objective is their average function  $f$ . Without loss of generality, we assume  $f_{1}$  is the master node and the others are local nodes. In each round, every local node can communicate with the master node certain information, such as the local parameter  $x$ , local gradient  $\nabla f_{i}(\boldsymbol{x})$ , and some global information gathered at the master node. Such a scheme can also be viewed as decentralized optimization over a star network [55].
Following the wisdom of statistical similarity residing in the data at different nodes, many previous works study scenarios where the individual functions exhibit relationships or, more specifically, certain homogeneity shared among the local  $f_{i}$ 's and  $f$ . The most common one is under the  $\delta$ -second-order
$^{\dagger}$ Academy for Advanced Interdisciplinary Studies; Peking University; lindachao@pku.edu.cn;
$^{\ddagger}$ School of Mathematical Sciences; Peking University; hanyuze97@pku.edu.cn;
$^{\S}$ Corresponding Author; School of Management; Xi'an Jiaotong University; SGIT AI Lab, State Grid Corporation of China; yehaishan@xjtu.edu.cn;
$^{1}$ School of Mathematical Sciences; Peking University; zhzhang@math.pku.edu.cn.
similarity assumption [33, 35], that is,
$$
\left| \left| \nabla^ {2} f _ {i} (\boldsymbol {x}) - \nabla^ {2} f (\boldsymbol {x}) \right| \right| \leq \delta , \forall \boldsymbol {x} \in \mathbb {R} ^ {d}, i \in \{1, \dots , n \}.
$$
Such an assumption also has different names in the literature, such as  $\delta$ -related assumption, bounded Hessian dissimilarity, or function similarity [7, 32, 51, 54, 62]. The rigorous definitions are deferred to Section 2. Moreover, the second-order similarity assumption can hold with a relatively small  $\delta$  compared to the smoothness coefficient of  $f_{i}$ 's in many practical settings, such as statistical learning. More insights on this can be found in the discussion presented in [54, Section 2]. The similarity assumption indicates that the data across different clients share common information on the second-order derivative, potentially leading to a reduction in communication among clients. Meanwhile, the cost of communication is often much higher than that of local computation in distributed optimization settings [9, 44, 30]. Hence, researchers are motivated to develop efficient algorithms characterized by low communication complexity, which is the primary objective of this paper as well.
Furthermore, we need to emphasize that prior research [25, 56, 63, 19, 43, 5] has shown tightly matched lower and upper bounds on computation complexity for the finite-sum objective in Eq. (1). These works focus on gradient complexity under (average) smoothness [63] instead of communication complexity under similarity. Indeed, we will also discuss and compare the gradient complexity as shown in [35], to explore the trade-off between communication and gradient complexity.
Although the development of distributed optimization with similarity has lasted for years, the optimal complexity under full participation was only recently achieved by Kovalev et al. [35]. They employed gradient-sliding [37] and obtained the optimal communication complexity  $\tilde{\mathcal{O}}(n\sqrt{\delta/\mu})$  for  $\mu$ -strongly convex  $f$  and  $\delta$ -related  $f_i$ 's in Eq. (1). However, the full participation model requires the calculation of the whole gradient  $\nabla f(\cdot)$ , which incurs a communication cost of  $n-1$  in each round. In contrast, partial participation could reduce the communication burden and yield improved complexity. Hence, Khaled and Jin [33] introduced client sampling, a technique that selects one client for updating in each round. They developed a non-accelerated algorithm SVRP, which achieves the communication complexity of  $\tilde{\mathcal{O}}(n + \delta^2/\mu^2)$ . Additionally, they proposed a Catalyzed version of SVRP with the complexity  $\tilde{\mathcal{O}}(n + n^{3/4}\sqrt{\delta/\mu})$ , which is better than the rates obtained in the full participation setting.
We believe there are several potential avenues for improvement inspired by [33]. 1) Khaled and Jin [33] introduced the requirement that each individual function is strongly convex (see [33, Assumption 2]). However, this constraint is absent in prior works. Notably, in the context of full participation, even non-convexity is deemed acceptable<sup>6</sup>. A prominent example is the shift-and-invert approach to solving PCA [52, 23], where each component is smooth and non-convex, but the average function remains convex. Thus we doubt the necessity of requiring strong convexity for individual components. 2) In hindsight, it seems that the directly accelerated SVRP could only achieve a bound of  $\tilde{\mathcal{O}}(n + \sqrt{n} \cdot \delta / \mu)$  based on the current analysis, which is far from being satisfactory compared to its Catalyzed version. Consequently, there might be room for the development of a more effective algorithm for direct acceleration. 3) It is essential to note that the Catalyst framework introduces an additional log term in the overall complexity, along with the challenge of parameter tuning. This aspect is discussed in detail in [6, Section 1.2]. Therefore, we intend to address the aforementioned concerns, particularly on designing directly accelerated methods under the second-order similarity assumption.
In this paper, we address the above concerns under the average similarity condition. Our contributions are presented in detail below and we provide a comparison with previous works in Table 1:
- First, we combine gradient sliding and client sampling techniques to develop an improved non-accelerated algorithm named SVRS (Algorithms 1). SVRS achieves a communication complexity of  $\tilde{\mathcal{O}}(n + \sqrt{n} \cdot \delta / \mu)$ , surpassing SVRP in ill-conditioned cases. Notably, this rate does not need component strong convexity and applies to the function value gap instead of the parameter distance.
- Second, building on SVRS, we employ a classical interpolation framework motivated by Katyusha X [6] to introduce the directly accelerated SVRS (AccSVRS, Algorithm 2).
Readers can check that the proof of [35] only requires  $f_{1}(\cdot) + \frac{1}{2\theta}\| \cdot \|^{2}$  is strongly convex, which can be guaranteed by  $\delta$ -second-order similarity since  $f$  is  $\mu$ -strongly convex and  $\theta = 1 / (2\delta)$  therein.
Table 1: Comparison of communication under similarity for the strongly convex objective.
Method/Reference	Communication complexity	Assumptions
No Sampling	AccExtragradient [35]	O(n√δ/μ log 1/ε)	SS only for fi
Lower bound [7]	Ω(n√δ/μ log 1/ε)	SS for fi's
Client Sampling	SVRP [33]	O((n+δ2/μ2) log 1/ε)(1)	SC for fi's, AveSS
Catalyzed SVRP [33]	O((n+n3/4√δ/μ) log 1/ε log L/μ)(2)	SC for fi's, AveSS
SVRS (Thm 3.3)	O((n+√n·δ/μ) log 1/ε)	AveSS
AccSVRS (Thm 3.6)	O((n+n3/4√δ/μ) log 1/ε)	AveSS
Lower bound (Thm 4.4)	Ω(n+n3/4√δ/μ log 1/ε)(3)	AveSS
(1) The rate only applies to  $\mathbb{E}\left\| \pmb{x}_k - \pmb{x}_*\right\|^2$ , otherwise it would introduce  $L$  in the log term; (2) The term  $\log (L / \mu)$  comes from the Catalyst framework. See Appendix C for the detail. (3) Here we only list the rates of the common ill-conditioned case:  $\mu = \mathcal{O}(\delta /\sqrt{n})$ . See Appendices for the remaining case. Notation:  $\delta$ -similarity parameter (both for SS and AveSS),  $L$ -smoothness constant of  $f$ ,  $\mu$ -strong convexity constant of  $f$  (or  $f_i$ 's),  $\varepsilon$ -error of the solution for  $\mathbb{E}f(\pmb{x}_k) - f(\pmb{x}_*)$ . Here  $L\geq \delta \geq \mu \gg \epsilon >0$ . Abbreviation: SC=strong convexity, SS=second-order similarity, AveSS=average SS.
AccSVRS achieves the same communication bound of  $\tilde{\mathcal{O}}(n + n^{3/4}\sqrt{\delta/\mu})$  as Catalyzed SVRP. Specifically, our bound is entirely smoothness-free and slightly outperforms Catalyzed SVRP, featuring a log improvement and not requiring component strong convexity.
- Third, by considering the proximal incremental first-order oracle in the centralized distributed framework, we establish a lower bound, which nearly matches the upper bound of AccSVRS in ill-conditioned cases.
Gradient sliding/Oracle Complexity Separation. For optimization problems with a separated structure or multiple building blocks, such as Eq. (1), there are scenarios where computing the gradients/values of some parts (or the whole) is more expensive than the others (or a partial one). In response to this challenge, techniques such as the gradient-sliding method [37] and the concept of oracle complexity separation [28] have emerged. These methods advocate for the infrequent use of more expensive oracles compared to their less resource-intensive counterparts. This strategy has found applications in zero-order [12, 21, 28, 53], first-order [37-39, 31] and high-order methods [31, 24, 3], as well as in addressing saddle point problems [4, 13]. Our algorithms can be viewed as a variance-reduced version of gradient sliding tailored to leverage the similarity assumption.
Distributed optimization under similarity. Distributed optimization has a long history with a plethora of existing works and surveys. To streamline our discussion, we only list the most relevant references, particularly under the similarity and strong convexity assumptions. In the full participation setting, which involves deterministic methods, the first algorithm credits to DANE [51], though its analysis is limited to quadratic objectives. Subsequently, AIDE [50], DANE-LS and DANE-HB [58] improved the rates for quadratic objective; disco [62] SPAG [27], ACN [1] and DiRegINA [18] improved the rates for self-concordant objectives. As for general strongly convex objectives, Sun et al. [54] introduced the SONATA algorithm, and Tian et al. [55] proposed accelerated SONATA. However, their complexity bounds include additional log factors. These factors have recently been removed by Accelerated Extragradient [35], whose complexity bound perfectly matches the lower bound in [7]. We highly recommend the comparison of rates in [35, Table 1] for a comprehensive overview. Once the discussion of deterministic methods is concluded, Khaled and Jin [33] shifted their focus to stochastic methods using client sampling. They proposed SVRP and its Catalyzed version, both of which exhibited superior rates compared to deterministic methods.
Notation. We denote vectors by lowercase bold letters (e.g.,  $\boldsymbol{w},\boldsymbol{x}$ ), and matrices by capital bold letters (e.g.,  $\mathbf{A},\mathbf{B}$ ). We let  $\| \cdot \|$  be the  $\ell_2$ -norm for vectors, or induced  $\ell_2$ -norm for a given matrix:  $\| \mathbf{A} \| = \sup_{\boldsymbol{u} \neq 0} \| \mathbf{A} \boldsymbol{u} \| / \| \boldsymbol{u} \|$ . We abbreviate  $[n] = \{1,\dots,n\}$  and  $\mathbf{I}_d \in \mathbb{R}^{d \times d}$  is the identity matrix. We use  $\mathbf{0}$  for the all-zero vector/matrix, whose size will be specified by a subscript, if necessary, and otherwise is clear from the context. We denote  $\mathrm{Unif}(S)$  as the uniform distribution over set  $S$ . We say  $T \sim \operatorname{Geom}(p)$  for  $p \in (0,1]$  if  $\mathbb{P}(T = k) = (1 - p)^{k-1} p, \forall k \in \{1,2,\ldots\}$ , i.e.,  $T$  obeys a geometric distribution. We adopt  $\mathbb{E}_k$  as the expectation for all randomness appeared in step  $k$ , and  $\mathbb{1}_A$  as the indicator function on event  $A$ , i.e.,  $\mathbb{1}_A = 1$  if event  $A$  holds, and 0 otherwise. We use  $\mathcal{O}(\cdot), \Omega(\cdot), \Theta(\cdot)$  and  $\tilde{\mathcal{O}}(\cdot)$  notation to hide universal constants and log-factors. We define the Bregman divergence induced by a differentiable (convex) function  $h: \mathbb{R}^d \to \mathbb{R}$  as  $D_h(\boldsymbol{x},\boldsymbol{y}) := h(\boldsymbol{x}) - h(\boldsymbol{y}) - \langle \nabla h(\boldsymbol{y}), \boldsymbol{x} - \boldsymbol{y} \rangle$ .
Definitions. We present the following common definitions used in this paper.
Definition 2.1 A differentiable function  $g \colon \mathbb{R}^d \to \mathbb{R}$  is  $\mu$ -strongly convex (SC) if
$$
g (\boldsymbol {y}) \geq g (\boldsymbol {x}) + \left\langle \nabla g (\boldsymbol {x}), \boldsymbol {y} - \boldsymbol {x} \right\rangle + \frac {\mu}{2} \left\| \boldsymbol {y} - \boldsymbol {x} \right\| ^ {2}, \forall \boldsymbol {x}, \boldsymbol {y} \in \mathbb {R} ^ {d}. \tag {2}
$$
Particularly, if  $\mu = 0$ , we say that  $g$  is convex.
Definition 2.2 A differentiable function  $g \colon \mathbb{R}^d \to \mathbb{R}$  is  $L$ -smooth if
$$
g (\boldsymbol {y}) \leq g (\boldsymbol {x}) + \left\langle \nabla g (\boldsymbol {x}), \boldsymbol {y} - \boldsymbol {x} \right\rangle + \frac {L}{2} \| \boldsymbol {y} - \boldsymbol {x} \| ^ {2}, \forall \boldsymbol {x}, \boldsymbol {y} \in \mathbb {R} ^ {d}. \tag {3}
$$
There are many basic inequalities involving strong convexity and smoothness, see [22, Appendix A.1] for an introduction. Next, we present the definition of second-order similarity in distributed optimization.
Definition 2.3 The differentiable functions  $f_{i}$ 's satisfy  $\delta$ -average second-order similarity (AveSS) if the following inequality holds for  $f_{i}$ 's and  $f = \frac{1}{n}\sum_{i=1}^{n}f_{i}$ :
$$
(A v e S S) \quad \frac {1}{n} \sum_ {i = 1} ^ {n} \| [ \nabla [ f _ {i} - f ] (\boldsymbol {x}) - \nabla [ f _ {i} - f ] (\boldsymbol {y}) ] \| ^ {2} \leq \delta^ {2} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2}, \forall \boldsymbol {x}, \boldsymbol {y} \in \mathbb {R} ^ {d}. \tag {4}
$$
Definition 2.4 The differentiable functions  $f_{i}$ 's satisfy  $\delta$ -component second-order similarity (SS) if the following inequality holds for  $f_{i}$ 's and  $f = \frac{1}{n}\sum_{i=1}^{n}f_{i}$ :
$$
\| [ \nabla [ f _ {i} - f ] (\boldsymbol {x}) - \nabla [ f _ {i} - f ] (\boldsymbol {y}) ] \| ^ {2} \leq \delta^ {2} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2}, \forall \boldsymbol {x}, \boldsymbol {y} \in \mathbb {R} ^ {d}, i \in [ n ]. \tag {5}
$$
Definitions 2.3 and 2.4 first appear in [33], which is an analogy to (average) smoothness in prior literature [63]. Particularly,  $f_{i}$ 's satisfy  $\delta$ -AveSS implies that  $(f - f_{i})$ 's satisfy  $\delta$ -average smoothness, while  $f_{i}$ 's satisfy  $\delta$ -SS implies that  $(f - f_{i})$ 's satisfy  $\delta$ -smoothness. Additionally, many researchers [32, 7, 51, 62, 54, 35] use the equivalent one defined by Hessian similarity (HS) if assuming that  $f_{i}$ 's are twice differentiable. Thus we also list them below and leave the derivation in Appendix B.
$$
\left(\operatorname {A v e H S}\right) \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \nabla^ {2} f _ {i} (\boldsymbol {x}) - \nabla^ {2} f (\boldsymbol {x}) \right] ^ {2} \right\| \leq \delta^ {2}; (\mathrm {H S}) \left\| \nabla^ {2} f _ {i} (\boldsymbol {x}) - \nabla^ {2} f (\boldsymbol {x}) \right\| \leq \delta , \forall i \in [ n ]. \tag {6}
$$
Since our algorithm is a first-order method, we adopt the gradient description of similarity (Definitions 2.3 and 2.4) without assuming twice differentiability for brevity.
As mentioned in [7, 54], if  $f_{i}$ 's satisfy  $\delta$ -AveSS (or SS), and  $f$  is  $\mu$ -strongly convex and  $L$ -smooth, then generally  $L \gg \delta \gg \mu > 0$  for large datasets in practice. Therefore, researchers aim to develop algorithms that achieve communication complexity solely related to  $\delta, \mu$  (or log terms of  $L$ ). This is also our objective. To finish this section, we will clarify several straightforward yet essential propositions, and the proofs are deferred to Appendix A.
Proposition 2.5 We have the following properties among SS, AveSS, and SC: 1)  $\delta$ -SS implies  $\delta$ -AveSS, but  $\delta$ -AveSS only implies  $\sqrt{n}\delta$ -SS. 2) If  $f_{i}$ 's satisfy  $\delta$ -SS and  $f$  is  $\mu$ -strongly convex, then for all  $i \in [n]$ ,  $f_{i}(\cdot) + \frac{\delta - \mu}{2} \| \cdot \|^{2}$  is convex, i.e.,  $f_{i}$  is  $(\delta - \mu)$ -almost convex [14].
3 Algorithm and Theory
In this section, we introduce our main algorithms, which are developed to solve the distributed optimization problem in Eq. (1) under Assumption 1 below:
Assumption 1 We assume that  $f_{i}$ 's satisfy  $\delta$ -AveSS, and  $f$  is  $\mu$ -strongly convex with  $\delta \geq \mu > 0$ .
Assumption 1 does not need each  $f_{i}$  to be  $\mu$ -strongly convex. In fact, it is acceptable that  $f_{i}$ 's are non-convex, since by Proposition 2.5,  $f_{i}$ 's are  $(\sqrt{n}\delta -\mu)$ -almost convex [14]. In the following, we first propose our new algorithm SVRS, which combines the techniques of gradient sliding and variance reduction, resulting in improved rates. Then we establish the directly accelerated method motivated by [6].
3.1 No Acceleration Version: SVRS
We first show the one-epoch Stochastic Variance-Reduced Sliding (SVRS $^{1ep}$ ) method in Algorithm 1. Before delving into the theoretical analysis, we present some key insights into our method. These insights aim to enhance comprehension and facilitate connections with other algorithms.
Variance Reduction. Our algorithm can be viewed as adding variance reduction from [35]. Besides the acceleration step, the main difference lies in the proximal step, where Kovalev et al. [35] solved:
$$
\boldsymbol {x} _ {t + 1} \approx \underset {\boldsymbol {x} \in \mathbb {R} ^ {d}} {\arg \min } B _ {\theta} ^ {t} (\boldsymbol {x}) := \left\langle \nabla f (\boldsymbol {x} _ {t}) - \nabla f _ {1} (\boldsymbol {x} _ {t}), \boldsymbol {x} - \boldsymbol {x} _ {t} \right\rangle + \frac {1}{2 \theta} \left\| \boldsymbol {x} - \boldsymbol {x} _ {t} \right\| ^ {2} + f _ {1} (\boldsymbol {x}).
$$
To save the heavy communication burden of calculating  $\nabla f(\pmb{x}_t)$ , we apply client sampling by selecting a random  $\nabla f_{i_t}(\pmb{x}_t)$  in the  $t$ -th step. However, this substitution introduces significant noise. To mitigate this, we incorporate a correction term  $\pmb{g}_t = \nabla f_{i_t}(\pmb{w}_0) - \nabla f(\pmb{w}_0)$  from previous wisdom [29] to reduce the variance.
Gradient sliding. Our algorithm can be viewed as adding gradient sliding from SVRP [33]. The main difference also lies in the proximal point problem, where Khaled and Jin [33] solved:
$$
\boldsymbol{x}_{t + 1}\approx \operatorname *{arg  min}_{\boldsymbol {x}\in \mathbb{R}^{d}}C_{\theta}^{t}(\boldsymbol {x}):= \langle -\boldsymbol{g}_{t},\boldsymbol {x} - \boldsymbol{x}_{t}\rangle +\frac{1}{2\theta}\left\| \boldsymbol {x} - \boldsymbol{x}_{t}\right\|^{2} + f_{i_{t}}(\boldsymbol {x}).
$$
Here we adopt a fixed proximal function  $f_{1}$  instead of  $f_{i_t}$ , which can be viewed as approximating  $f_{i_t}(\pmb{x}) \approx f_1(\pmb{x}) + [f_{i_t} - f_1](\pmb{x}_t) + \langle \nabla [f_{i_t} - f_1](\pmb{x}_t), \pmb{x} - \pmb{x}_t \rangle + \frac{1}{2\theta'} \| \pmb{x} - \pmb{x}_t \|^2$  with a properly chosen  $\theta' > 0$ . Such a modification is motivated by [35], where they reformulated the objective as  $f(\pmb{x}) = [f(\pmb{x}) - f_1(\pmb{x})] + f_1(\pmb{x})$ . Thus they could employ gradient sliding to skip heavy computations of  $\nabla [f - f_1](\pmb{x})$  by utilizing the easy computations of  $\nabla f_{1}(\pmb{x})$  more times. Fixing the proximal function  $f_{1}$  leads to the same metric space owned by  $f_{1}$  in each step, which could benefit the analysis and alleviate the requirements on  $f_{i}$ 's compared to SVRP. Indeed, in our setting  $f_{1}$  can be replaced by any other fixed client  $f_{b}, b \in [n]$ . In this case, the master node would be  $f_{b}$  instead of  $f_{1}$ .
Bregman-SVRG. Our algorithm can be viewed as the classical Bregman-SVRG [20] with the reference function  $f_{1}(\cdot) + \frac{1}{2\theta}\| \cdot \|^{2}$  after introducing the Bregman divergence:
$$
\boldsymbol {x} _ {t + 1} \approx \underset {\boldsymbol {x} \in \mathbb {R} ^ {d}} {\arg \min } A _ {\theta} ^ {t} (\boldsymbol {x}) \overset {(7)} {=} \underset {\boldsymbol {x} \in \mathbb {R} ^ {d}} {\arg \min } \langle \nabla f _ {i _ {t}} (\boldsymbol {x} _ {t}) - \nabla [ f _ {i _ {t}} - f ] (\boldsymbol {w} _ {0}), \boldsymbol {x} - \boldsymbol {x} _ {t} \rangle + D _ {f _ {1} (\cdot) + \frac {1}{2 \theta} \| \cdot \| ^ {2}} (\boldsymbol {x}, \boldsymbol {x} _ {t}).
$$
We need to emphasize that the proof of Bregman-SVRG requires additional structural assumptions [20, Assumption 3], which is not directly applicable in our setting. Hence, the rigorous proof of Bregman-SVRG under our similarity assumption is still meaningful as far as we are concerned.
3.1.1 Communication Complexity under Distributed Settings
When applied to the distributed system, the communication complexity of SVRS<sup>1ep</sup> can be described as follows: At the beginning of each epoch, the master (corresponding to  $f_{1}$ ) sends  $\boldsymbol{w}_{0}$  to all clients. Each client computes  $\nabla f_{i}(\boldsymbol{w}_{0})$  from its local data and sends it back to the master. The master then builds  $\nabla f(\boldsymbol{w}_{0})$  after collecting all  $\nabla f_{i}(\boldsymbol{w}_{0})$ 's. The communication complexity is  $2(n - 1)$  in this
Algorithm 1 SVRS $^{1\mathrm{ep}}(f, \boldsymbol{w}_0, \theta, p)$
1: Input:  $\pmb{w}_0 \in \mathbb{R}^d, p \in (0, 1), \theta > 0$
2: Initialize  $\pmb{x}_0 = \pmb{w}_0$ , compute  $\nabla f(\pmb{w}_0)$ , and set  $T \sim \mathrm{Geom}(p)$
3: for  $t = 0,1,2,\ldots ,T - 1$  do
4: Sample  $i_t \sim \mathrm{Unif}([n])$  and compute  $\pmb{g}_t = \nabla f_{i_t}(\pmb{w}_0) - \nabla f(\pmb{w}_0)$
5: Approximately solve the local proximal point problem:
$$
\boldsymbol {x} _ {t + 1} \approx \underset {\boldsymbol {x} \in \mathbb {R} ^ {d}} {\arg \min } A _ {\theta} ^ {t} (\boldsymbol {x}) := \left\langle \nabla f _ {i _ {t}} \left(\boldsymbol {x} _ {t}\right) - \nabla f _ {1} \left(\boldsymbol {x} _ {t}\right) - \boldsymbol {g} _ {t}, \boldsymbol {x} - \boldsymbol {x} _ {t} \right\rangle + \frac {1}{2 \theta} \| \boldsymbol {x} - \boldsymbol {x} _ {t} \| ^ {2} + f _ {1} (\boldsymbol {x}) \tag {7}
$$
6: end for
7: Output:  $x_{T}$