SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, and compressed communication. We prove that the new method has optimal oracle complexity and state-of-the-art communication complexity in the partial participation setting. Regardless of the communication compression feature, our method successfully combines variance reduction and partial participation: we get the optimal oracle complexity, never need the participation of all nodes, and do not require the bounded gradients (dissimilarity) assumption.
Federated and distributed learning have become very popular in recent years (Konečný et al., 2016; McMahan et al., 2017). The current optimization tasks require much computational resources and machines. Such requirements emerge in machine learning, where massive datasets and computations are distributed between cluster nodes (Lin et al., 2017; Ramesh et al., 2021). In federated learning, nodes, represented by mobile phones, laptops, and desktops, do not send their data to a server due to privacy and their huge number (Ramaswamy et al., 2019), and the server remotely orchestrates the nodes and communicates with them to solve an optimization problem.
As in classical optimization tasks, one of the main current challenges is to find computationally efficient optimization algorithms. However, the nature of distributed problems induces many other (Kairouz et al., 2021), including i) partial participation of nodes in algorithm steps: due to stragglers (Li et al., 2020) or communication delays (Vogels et al., 2021), ii) communication bottleneck: even if a node participates, it can be costly to transmit information to a server or other nodes (Alistarh et al., 2017; Ramesh et al., 2021; Kairouz et al., 2021; Sapio et al., 2019; Narayanan et al., 2019). It is necessary to develop a method that considers these problems.
2 Optimization Problem
Let us consider the nonconvex distributed optimization problem
where  $f_{i}:\mathbb{R}^{d}\to \mathbb{R}$  is a smooth nonconvex function for all  $i\in [n]\coloneqq \{1,\ldots ,n\}$ . The full information about function  $f_{i}$  is stored on  $i^{\mathrm{th}}$  node. The communication between nodes is maintained in the parameters server fashion (Kairouz et al., 2021): we have a server that receives compressed
information from nodes, updates a state, and broadcasts an updated model. Since we work in the nonconvex world, our goal is to find an  $\varepsilon$ -solution ( $\varepsilon$ -stationary point) of (1): a (possibly random) point  $\widehat{x} \in \mathbb{R}^d$ , such that  $\operatorname{E}\left[\|\nabla f(\widehat{x})\|^2\right] \leq \varepsilon$ .
We consider three settings:
1. Gradient Setting. The  $i^{\mathrm{th}}$  node has only access to the gradient  $\nabla f_{i}:\mathbb{R}^{d}\to \mathbb{R}^{d}$  of function  $f_{i}$ . Moreover, the following assumptions for the functions  $f_{i}$  hold.
Assumption 1. There exists  $f^{*} \in \mathbb{R}$  such that  $f(x) \geq f^{*}$  for all  $x \in \mathbb{R}$ .
Assumption 2. The function  $f$  is  $L$ -smooth, i.e.,  $\| \nabla f(x) - \nabla f(y) \| \leq L \| x - y \|$  for all  $x, y \in \mathbb{R}^d$ .
Assumption 3. The functions  $f_{i}$  are  $L_{i}$ -smooth for all  $i \in [n]$ . Let us define  $\widehat{L}^2 \coloneqq \frac{1}{n}\sum_{i = 1}^{n}L_i^2$ .
2. Finite-Sum Setting. The functions  $\{f_i\}_{i=1}^n$  have the finite-sum form
$$
f _ {i} (x) = \frac {1}{m} \sum_ {j = 1} ^ {m} f _ {i j} (x), \quad \forall i \in [ n ], \tag {2}
$$
where  $f_{ij}:\mathbb{R}^d\to \mathbb{R}$  is a smooth nonconvex function for all  $j\in [m]$ .
We assume that Assumptions 1, 2 and 3 hold and the following assumption.
Assumption 4. The function  $f_{ij}$  is  $L_{ij}$ -smooth for all  $i \in [n], j \in [m]$ . Let  $L_{\max} := \max_{i \in [n], j \in [m]} L_{ij}$ .
3. Stochastic Setting. The function  $f_{i}$  is an expectation of a stochastic function,
$$
f _ {i} (x) = \mathrm {E} _ {\xi} \left[ f _ {i} (x; \xi) \right], \quad \forall i \in [ n ], \tag {3}
$$
where  $f_{i}:\mathbb{R}^{d}\times \Omega_{\xi}\to \mathbb{R}$ . For a fixed  $x\in \mathbb{R}$ ,  $f_{i}(x;\xi)$  is a random variable over some distribution  $\mathcal{D}_i$ , and, for a fixed  $\xi \in \Omega_{\xi}$ ,  $f_{i}(x;\xi)$  is a smooth nonconvex function. The  $i^{\mathrm{th}}$  node has only access to a stochastic gradients  $\nabla f_{i}(\cdot ;\xi_{ij})$  of the function  $f_{i}$  through the distribution  $\mathcal{D}_i$ , where  $\xi_{ij}$  is a sample from  $\mathcal{D}_i$ . We assume that Assumptions 1, 2 and 3 hold and the following assumptions.
Assumption 5. For all  $i \in [n]$  and for all  $x \in \mathbb{R}^d$ , the stochastic gradient  $\nabla f_i(x;\xi)$  is unbiased and has bounded variance, i.e.,  $\mathrm{E}_{\xi}\left[\nabla f_i(x;\xi)\right] = \nabla f_i(x)$ , and  $\mathrm{E}_{\xi}\left[\| \nabla f_i(x;\xi) - \nabla f_i(x)\|^2\right] \leq \sigma^2$ , where  $\sigma^2 \geq 0$ .
Assumption 6. For all  $i \in [n]$  and for all  $x, y \in \mathbb{R}$ , the stochastic gradient  $\nabla f_i(x; \xi)$  satisfies the mean-squared smoothness property, i.e.,  $\mathrm{E}_{\xi} \left[ \| \nabla f_i(x; \xi) - \nabla f_i(y; \xi) \|^2 \right] \leq L_{\sigma}^2 \| x - y \|^2$ .
We compare algorithms using the oracle complexity, i.e., the number of (stochastic) gradients that each node has to calculate to get  $\varepsilon$ -solution, and the communication complexity, i.e., the number of bits that each node has to send to the server to get  $\varepsilon$ -solution.
2.1 Unbiased Compressors
We use the concept of unbiased compressors to alleviate the communication bottleneck. The unbiased compressors quantize and/or sparsify vectors that the nodes send to the server.
Definition 1. A stochastic mapping  $\mathcal{C}:\mathbb{R}^d\to \mathbb{R}^d$  is an unbiased compressor if there exists  $\omega \in \mathbb{R}$
$$
\text {s u c h} \quad \mathrm {E} [ \mathcal {C} (x) ] = x \quad \text {a n d} \quad \mathrm {E} \left[ \| \mathcal {C} (x) - x \| ^ {2} \right] \leq \omega \| x \| ^ {2} \quad \forall x \in \mathbb {R} ^ {d}. \tag {4}
$$
We denote a set of stochastic mappings that satisfy Definition 1 as  $\mathbb{U}(\omega)$ . In our methods, the nodes make use of unbiased compressors  $\{\mathcal{C}_i\}_{i=1}^n$ . The community developed a large number of unbiased compressors, including RandK (see Definition 5) (Beznosikov et al., 2020; Stich et al., 2018), Adaptive sparsification (Wangni et al., 2018) and Natural compression and dithering (Horváth et al., 2019a). We are aware of correlated compressors by Szlendak et al. (2021) and quantizers by Suresh et al. (2022) that help in the homogeneous regimes, but in this work, we are mainly concentrated on generic heterogeneous regimes, though, for simplicity, assume the independence of the compressors.
Assumption 7.  $\mathcal{C}_i \in \mathbb{U}(\omega)$  for all  $i \in [n]$ , and the compressors are statistically independent.
<sup>1</sup>Note that this strategy can be used in peer-to-peer communication, assuming that the server is an abstraction and all its algorithmic steps are performed on each node.
2Note that  $L\leq \widehat{L}$ $\widehat{L}\leq L_{\mathrm{max}}$  , and  $\widehat{L}\leq L_{\sigma}$
Table 1: Summary of methods that solve the problem (1) in the stochastic setting (3). Abbr.: VR (Variance Reduction) = Does a method have the optimal oracle complexity  $\mathcal{O}\left(\frac{\sigma^2}{\varepsilon} + \frac{\sigma}{\varepsilon^{3/2}}\right)$ ? PP (Partial Participation) = Does a method support partial participation from Section 2.2? CC = Does a method have the communication complexity equals to  $\mathcal{O}\left(\frac{\omega}{\sqrt{n}\varepsilon}\right)$ ?
Method	VR	PP	CC	Limitations
SPIDER, SARAH, PAGE, STORM(Fang et al., 2018; Nguyen et al., 2017)(Li et al., 2021a; Cutkosky and Orabona, 2019)	✓	✗	✗	-
MARINA(Gorbunov et al., 2021)	✓	X(a)	✓(b)	Suboptimal convergence rate(see (Tyurin and Richtárik, 2023)).
FedPAGE(Zhao et al., 2021b)	✗	X(a)	✗	Suboptimal oracle complexity O(σ2/ε2).
FRECON(Zhao et al., 2021a)	✗	✓	✓	-
FedAvg(McMahan et al., 2017; Karimireddy et al., 2020b)	✗	✓	✗	Bounded gradients (dissimilarity) assumption of fi.
SCAFFOLD(Karimireddy et al., 2020b)	✗	✓	✗	Suboptimal convergence rate(e).
MIME(c)(Karimireddy et al., 2020a)	X(d)	✓	✗	Calculates full gradient.Bounded gradients (dissimilarity) assumption of fi.Suboptimal oracle compl. O(1/ε3/2) in the setting (2).
CE-LSGD (for Partial Participation)(c)(Patel et al., 2022) (concurrent work)	✓	✓	✗	Bounded gradients (dissimilarity) assumption of fi.Suboptimal oracle compl. O(1/ε3/2) in the setting (2).
DASHA(Tyurin and Richtárik, 2023)	✓	✗or	✓	-
✗	✓	✓
DASHA-PP(new)	✓	✓	✓	-
(a) MARINA and FedPAGE, with a small probability, require the participation of all nodes so that they can not support partial participation from Section 2.2. Moreover, these methods provide suboptimal oracle complexities.
(b) On average, MARINA provides the compressed communication mechanism with complexity  $\mathcal{O}\left(\frac{\omega}{\sqrt{n\varepsilon}}\right)$ . However, with a small probability, this method sends non-compressed vectors.
(c) Note that MIME and CE-LSGD can not be directly compared with DASHA-PP because MIME and CE-LSGD consider the online version of the problem (1), and require more strict assumptions.
(d) Although MIME obtains the convergence rate  $\mathcal{O}\left(\frac{1}{\varepsilon^{3/2}}\right)$  of a variance reduced method, it requires the calculation of the full (exact) gradients.
(c) It can be seen when  $\sigma^2 = 0$ . Consider the  $s$ -nice sampling of the nodes, then SCAFFOLD requires  $\mathcal{O}\left(n^{3/2} / \varepsilon s^{3/2}\right)$  communication rounds to get  $\varepsilon$ -solution, while DASHA-PP requires  $\mathcal{O}\left(\sqrt{n} / \varepsilon s\right)$  communication rounds (see Theorem 4 with  $\omega = 0$ ,  $b = p_{\mathrm{a}} / 2 - p_{\mathrm{a}}$ , and  $p_{\mathrm{a}} = \frac{s}{n}$ ).
2.2 Nodes Partial Participation Assumptions
We now try to formalize the notion of partial participation. Let us assume that we have  $n$  events  $\{i^{\mathrm{th}} \text{ node is participating}\}$  with the following properties.
Assumption 8. The partial participation of nodes has the following distribution: exists constants  $p_{\mathrm{a}} \in (0,1]$  and  $p_{\mathrm{aa}} \in [0,1]$ , such that
1. Prob  $(i^{\mathrm{th}}$  node is participating)  $= p_{\mathrm{a}}\quad \forall i\in [n]$
2. Prob  $(i^{\mathrm{th}}$  and  $j^{\mathrm{th}}$  nodes are participating)  $= p_{\mathrm{aa}}$ $\forall i\neq j\in [n]$
3.  $p_{\mathrm{aa}} \leq p_{\mathrm{a}}^{2},$  (5)
and these events from different communication rounds are independent.
We are not fighting for the full generality and believe that more complex sampling strategies can be considered in the analysis. For simplicity, we settle upon Assumption 8. Standard partial participation strategies, including  $s$ -nice sampling, where the server chooses uniformly  $s$  nodes without replacement  $(p_{\mathrm{a}} = s / n$  and  $p_{\mathrm{aa}} = s(s - 1) / n(n - 1))$ , and independent participation, where each
Table 2: Summary of methods that solve the problem (1) in the finite-sum setting (2). Abbr.: VR (Variance Reduction) = Does a method have the optimal oracle complexity  $\mathcal{O}\left(m + \frac{\sqrt{m}}{\varepsilon}\right)$ ? PP and CC are defined in Table 1.
Method	VR	PP	CC	Limitations
SPIDER, PAGE(Fang et al., 2018; Li et al., 2021a)	✓	✘	✘	—
MARINA(Gorbunov et al., 2021)	✓	\(x^{(a)}\)	\(\checkmark^{(b)}\)	Suboptimal convergence rate(see (Tyurin and Richtárik, 2023)).
ZeroSARAH(Li et al., 2021b)	✓	✓	✘	Only homogeneous regime, i.e., the functions \(f_i\) are equal.
FedPAGE(Zhao et al., 2021b)	✘	\(x^{(a)}\)	✘	Suboptimal oracle complexity \(\mathcal{O}\left(\frac{m}{\varepsilon}\right)\).
DASHA(Tyurin and Richtárik, 2023)	✓	✘	✓	—
DASHA-PP(new)	✓	✓	✓	—
(a),(b) : see Table 1.
node independently participates with probability  $p_{\mathrm{a}}$  (due to independence, we have  $p_{\mathrm{aa}} = p_{\mathrm{a}}^{2}$ ), satisfy Assumption 8. In the literature,  $s$ -nice sampling is one of the most popular strategies (Zhao et al., 2021a; Richtárik et al., 2021; Reddi et al., 2020; Konečný et al., 2016).
3 Motivation and Related Work
The main goal of our paper is to develop a method for the nonconvex distributed optimization that will include three key features: variance reduction of stochastic gradients, compressed communication, and partial participation. We now provide an overview of the literature (see also Table 1 and Table 2).
1. Variance reduction of stochastic gradients
It is important to consider finite-sum (2) and stochastic (3) settings because, in machine learning tasks, either the number of local functions  $m$  is huge or the functions  $f_{i}$  is an expectation of a stochastic function due to the batch normalization (Ioffe and Szegedy, 2015) or random augmentation (Goodfellow et al., 2016), and it is infeasible to calculate the full gradients analytically. Let us recall the results from the nondistributed optimization. In the gradient setting, the optimal oracle complexity is  $\mathcal{O}\left(1 / \varepsilon\right)$ , achieved by the vanilla gradient descent (GD) (Carmon et al., 2020; Nesterov, 2018). In the finite-sum setting and stochastic settings, the optimal oracle complexities are  $\mathcal{O}\left(m + \frac{\sqrt{m}}{\varepsilon}\right)$  and  $\mathcal{O}\left(\frac{\sigma^2}{\varepsilon} + \frac{\sigma}{\varepsilon^{3/2}}\right)$  (Fang et al., 2018; Li et al., 2021a; Arjevani et al., 2019), accordingly, achieved by methods SPIDER, SARAH, PAGE, and STORM from (Fang et al., 2018; Nguyen et al., 2017; Li et al., 2021a; Cutkosky and Orabona, 2019).
2. Compressed communication
In distributed optimization (Ramesh et al., 2021; Xu et al., 2021), lossy communication compression can be a powerful tool to increase the communication speed between the nodes and the server. Different types of compressors are considered in the literature, including unbiased compressors (Alistarh et al., 2017; Beznosikov et al., 2020; Szlendak et al., 2021), contractive (biased) compressors (Richtárik et al., 2021), 3PC compressors (Richtárik et al., 2022). We will focus on unbiased compressors because methods DASHA and MARINA (Tyurin and Richtárik, 2023; Szlendak et al., 2021; Gorbunov et al., 2021) that employ unbiased compressors provide the current theoretical state-of-the-art (SOTA) communication complexities.
Many methods analyzed optimization methods with the unbiased compressors (Alistarh et al., 2017; Mishchenko et al., 2019; Horváth et al., 2019b; Gorbunov et al., 2021; Tyurin and Richtárik, 2023). In the gradient setting, the methods MARINA and DASHA by Gorbunov et al. (2021) and Tyurin and Richtárik (2023) establish the current SOTA communication complexity, each method needs  $\frac{1 + \omega / \sqrt{n}}{\varepsilon}$  communication rounds to get an  $\varepsilon$ -solution. In the finite-sum and stochastic settings, the current
SOTA communication complexity is attained by the DASHA method, while maintaining the optimal oracle complexities  $\mathcal{O}\left(m + \frac{\sqrt{m}}{\varepsilon\sqrt{n}}\right)$  and  $\mathcal{O}\left(\frac{\sigma^2}{\varepsilon n} +\frac{\sigma}{\varepsilon^{3 / 2}n}\right)$  per node.
3. Partial participation
From the beginning of federated learning era, the partial participation has been considered to be the essential feature of distributed optimization methods (McMahan et al., 2017; Konečný et al., 2016; Kairouz et al., 2021). However, previously proposed methods have limitations: i) methods MARINA and FedPAGE from (Gorbunov et al., 2021; Zhao et al., 2021b) still require synchronization of all nodes with a small probability. ii) in the stochastic settings, methods FedAvg, SCAFFOLD, and FRECON with the partial participation mechanism (McMahan et al., 2017; Karimireddy et al., 2020b; Zhao et al., 2021a) provide results without variance reduction techniques from (Fang et al., 2018; Li et al., 2021a; Cutkosky and Orabona, 2019) and, therefore, get suboptimal oracle complexities. Note that FRECON and DASHA reduce the variance only from compressors (in the partial participation and stochastic setting). iii) in the finite-sum setting, the ZeroSARAH method by Li et al. (2021b) focuses on the homogeneous regime only (the functions  $f_{i}$  are equal). iv) The MIME method by Karimireddy et al. (2020a) and the CE-LSGD method (for Partial Participation) by the concurrent paper (Patel et al., 2022) consider the online version of the problem (1). Therefore, MIME and CE-LSGD (for Partial Participation) require stricter assumptions, including the bounded inter-client gradient variance assumption. In the finite-sum setting (2), MIME and CE-LSGD obtain a suboptimal oracle complexity  $\mathcal{O}\left(1 / \varepsilon^{3 / 2}\right)$  while, in the full participation setting, it is possible to get the complexity  $\mathcal{O}\left(1 / \varepsilon\right)$ .
4 Contributions
We propose a new method DASHA-PP for the nonconvex distributed optimization.
- As far as we know, this is the first method that includes three key ingredients of federated learning methods: variance reduction of stochastic gradients, compressed communication, and partial participation.
- Moreover, this is the first method that combines variance reduction of stochastic gradients and partial participation flawlessly: i) it gets the optimal oracle complexity ii) does not require the participation of all nodes iii) does not require the bounded gradients assumption of the functions  $f_{i}$ .
- We prove convergence rates and show that this method has the optimal oracle complexity and the state-of-the-art communication complexity in the partial participation setting. Moreover, in our work, we observe a nontrivial side-effect from mixing the variance reduction of stochastic gradients and partial participation. It is a general problem not related to our methods or analysis that we discuss in Section C.
- In Section A, we present experiments where we validate our theory and compare our new methods to previous ones.
5 Algorithm Description and Main Challenges Towards Partial Participation
We now present DASHA-PP (see Algorithm 1), a family of methods to solve the optimization problem (1). When we started investigating the problem, we took DASHA as a baseline method for two reasons: the family of algorithms DASHA provides the current state-of-the-art communication complexities in the non-partial participation setting, and, unlike MARINA, it does not send non-compressed gradients and does not synchronize all nodes. Let us briefly discuss the main idea of DASHA, its problem in the partial participation setting, and why the refinement of DASHA is not an exercise.
In fact, the original DASHA method supports the partial participation of nodes in the gradient setting. Since the nodes only do the following steps (see full algorithm in Algorithm 6):
$$
g _ {i} ^ {t + 1} = g _ {i} ^ {t} + \mathcal {C} _ {i} (\nabla f _ {i} (x ^ {t + 1}) - (1 - a) \nabla f _ {i} (x ^ {t}) - a g _ {i} ^ {t}). \tag {6}
$$
Algorithm 1 DASHA-PP
1: Input: starting point  $x^0 \in \mathbb{R}^d$ , stepsize  $\gamma > 0$ , momentum  $a \in (0,1]$ , momentum  $b \in (0,1]$ , probability  $p_{\mathrm{page}} \in (0,1]$  (only in DASHA-PP-PAGE), batch size  $B$  (only in DASHA-PP-PAGE, DASHA-PP-FINITE-MVR and DASHA-PP-MVR), probability  $p_{\mathrm{a}} \in (0,1]$  that a node is participating(a), number of iterations  $T \geq 1$
2: Initialize  $g_{i}^{0}\in \mathbb{R}^{d}$ $h_i^0\in \mathbb{R}^d$  on the nodes and  $g^0 = \frac{1}{n}\sum_{i = 1}^n g_i^0$  on the server
3: Initialize  $h_{ij}^{0} \in \mathbb{R}^{d}$  on the nodes and take  $h_i^0 = \frac{1}{m}\sum_{j = 1}^{m}h_{ij}^{0}$  (only in DASHA-PP-FINITE-MVR)
4: for  $t = 0,1,\ldots ,T - 1$  do
5:  $x^{t + 1} = x^t -\gamma g^t$
6: Broadcast  $x^{t+1}$ ,  $x^t$  to all participating $^{(\mathrm{a})}$  nodes
7: for  $i = 1,\dots ,n$  in parallel do
8: if  $i^{\mathrm{th}}$  node is participating(a) then
9: Calculate  $k_i^{t+1}$  using Algorithm 2, 3, 4 or 5
10:  $h_i^{t + 1} = h_i^t +\frac{1}{p_n} k_i^{t + 1}$
11:  $m_{i}^{t + 1} = \mathcal{C}_{i}\left(\frac{1}{p_{a}} k_{i}^{t + 1} - \frac{a}{p_{a}}\left(g_{i}^{t} - h_{i}^{t}\right)\right)$
12:  $g_{i}^{t + 1} = g_{i}^{t} + m_{i}^{t + 1}$
13: Send  $m_i^{t + 1}$  to the server
14: else
15:  $h_{ij}^{t + 1} = h_{ij}^{t}$  (only in DASHA-PP-FINITE-MVR)
16:  $h_i^{t + 1} = h_i^t$ $g_{i}^{t + 1} = g_{i}^{t}$ $m_{i}^{t + 1} = 0$
17: end if
18: end for
19:  $g^{t + 1} = g^t +\frac{1}{n}\sum_{i = 1}^n m_i^{t + 1}$
20: end for
21: Output:  $\hat{x}^T$  chosen uniformly at random from  $\{x^t\}_{k=0}^{T-1}$
(a): For the formal description see Section 2.2.
Algorithm 2 Calculate  $k_{i}^{t + 1}$  for DASHA-PP in the gradient setting. See line 9 in Alg. 1.
$$
1: k _ {i} ^ {t + 1} = \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) - b \left(h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t})\right)
$$
Algorithm 3 Calculate  $k_{i}^{t + 1}$  for DASHA-PP-PAGE in the finite-sum setting. See line 9 in Alg. 1
1: Generate a random set  $I_{i}^{t}$  of size  $B$  from  $[m]$  with replacement
$$
2: k _ {i} ^ {t + 1} = \left\{ \begin{array}{l} \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) - \frac {b}{p _ {\text {p a g e}}} \left(h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t})\right), \\ \text {w i t h p r o b a b i l i t y p _ {\text {p a g e}} o n a l l p a r t i c i p a t i n g n o d e s}, \\ \frac {1}{B} \sum_ {j \in I _ {i} ^ {t}} \left(\nabla f _ {i j} (x ^ {t + 1}) - \nabla f _ {i j} (x ^ {t})\right), \\ \text {w i t h p r o b a b i l i t y 1 - p _ {\text {p a g e}} o n a l l p a r t i c i p a t i n g n o d e s} \end{array} \right.
$$
Algorithm 4 Calc.  $k_{i}^{t + 1}$  for DASHA-PP-FINITE-MVR in the finite-sum setting. See line 9 in Alg. 1
1: Generate a random set  $I_{i}^{t}$  of size  $B$  from  $[m]$  without replacement
$$
\begin{array}{l} 2: k _ {i j} ^ {t + 1} = \left\{ \begin{array}{l l} \frac {m}{B} \left(\nabla f _ {i j} (x ^ {t + 1}) - \nabla f _ {i j} (x ^ {t}) - b \left(h _ {i j} ^ {t} - \nabla f _ {i j} (x ^ {t})\right)\right), & j \in I _ {i} ^ {t}, \\ 0, & j \not \in I _ {i} ^ {t} \end{array} \right. \\ 3: h _ {i j} ^ {t + 1} = h _ {i j} ^ {t} + \frac {1}{p _ {\mathrm {a}}} k _ {i j} ^ {t + 1} \\ 4: k _ {i} ^ {t + 1} = \frac {1}{m} \sum_ {j = 1} ^ {m} k _ {i j} ^ {t + 1} \\ \end{array}
$$
Algorithm 5 Calculate  $k_{i}^{t + 1}$  for DASHA-PP-MVR in the stochastic setting. See line 9 in Alg. 1
1: Generate i.i.d. samples  $\{\xi_{ij}^{t + 1}\}_{j = 1}^{B}$  of size  $B$  from  $\mathcal{D}_i$ .
$$
2: k _ {i} ^ {t + 1} = \frac {1}{B} \sum_ {j = 1} ^ {B} \nabla f _ {i} (x ^ {t + 1}; \xi_ {i j} ^ {t + 1}) - \frac {1}{B} \sum_ {j = 1} ^ {B} \nabla f _ {i} (x ^ {t}; \xi_ {i j} ^ {t + 1}) - b \left(h _ {i} ^ {t} - \frac {1}{B} \sum_ {j = 1} ^ {B} \nabla f _ {i} (x ^ {t}; \xi_ {i j} ^ {t + 1})\right)
$$
The partial participation mechanism (independent participation from Section 2.2) can be easily implemented here if we temporally redefine the compressor and use another one instead:
$$
\mathcal {C} _ {i} ^ {p _ {\mathrm {a}}} := \left\{ \begin{array}{l l} \frac {1}{p _ {\mathrm {a}}} \mathcal {C} _ {i}, \text {w . p .} p _ {\mathrm {a}}, & \stackrel {(6)} {\Rightarrow} g _ {i} ^ {t + 1} = \left\{ \begin{array}{l l} g _ {i} ^ {t} + \frac {1}{p _ {\mathrm {a}}} \mathcal {C} _ {i} \left(\nabla f _ {i} (x ^ {t + 1}) - (1 - a) \nabla f _ {i} (x ^ {t}) - a g _ {i} ^ {t}\right), \text {w . p .} p _ {\mathrm {a}} \\ 0, \text {w . p .} 1 - p _ {\mathrm {a}}. & g _ {i} ^ {t}, \end{array} \right. \\ 0, \text {w . p .} 1 - p _ {\mathrm {a}}. \end{array} \right.
$$
With probability  $1 - p_{\mathrm{a}}$ , a node does not update  $g_{i}^{t}$  and does not send anything to the server. The main observation is that we can do this trick since  $g_{i}^{t + 1}$  depends only on the vectors  $x^{t + 1}, x^{t}$ , and  $g_{i}^{t}$ . The points  $x^{t + 1}$  and  $x^{t}$  are only available in a node only during its participation.