SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, we focus our attention on partial participation in the finite-sum and stochastic settings. Consider the nodes' steps in DASHA-MVR (Tyurin and Richtárik, 2023) (see Algorithm 7) that is designed for the stochastic setting:
$$
h _ {i} ^ {t + 1} = \nabla f _ {i} \left(x ^ {t + 1}; \xi_ {i} ^ {t + 1}\right) + (1 - b) \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}; \xi_ {i} ^ {t + 1}\right)\right), \tag {7}
$$
$$
g _ {i} ^ {t + 1} = g _ {i} ^ {t} + \mathcal {C} _ {i} \left(h _ {i} ^ {t + 1} - h _ {i} ^ {t} - a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right). \tag {8}
$$
Now we have two sequences  $h_i^t$  and  $g_i^t$ . Even if we use the same trick for (8), we still have to update (7) in every iteration of the algorithm since  $g_i^{t + 1}$  additionally depends on  $h_i^{t + 1}$  and  $h_i^t$ . In other words, if a node does not update  $g_i^t$  and does not send anything to the server, it still has to update  $h_i^t$ , what is impossible without the points  $x^{t + 1}$  and  $x^t$ . One of the main challenges was to "guess" how to generalize (7) and (8) to the partial participation setting. We now provide a solution (DASHA-PP-MVR with the batch size  $B = 1$ ):
$$
\begin{array}{l} h _ {i} ^ {t + 1} = h _ {i} ^ {t} + \frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1}, k _ {i} ^ {t + 1} = \nabla f _ {i} (x ^ {t + 1}; \xi_ {i} ^ {t + 1}) - \nabla f _ {i} (x ^ {t}; \xi_ {i} ^ {t + 1}) - b \left(h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}; \xi_ {i} ^ {t + 1})\right), \\ g _ {i} ^ {t + 1} = g _ {i} ^ {t} + \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) \text {w i t h p r o b a b i l i t y} p _ {\mathrm {a}}, \tag {9} \\ \end{array}
$$
and  $h_i^{t + 1} = h_i^t$ ,  $g_i^{t + 1} = g_i^t$  with probability  $1 - p_{\mathrm{a}}$ .
Now both control variables  $g_{i}^{t}$  and  $h_{i}^{t}$  do not change with the probability  $1 - p_{\mathrm{a}}$ . When the  $i^{\mathrm{th}}$  node participates, the update rules of  $g_{i}^{t + 1}$  and  $h_{i}^{t + 1}$  in (9) were adapted to make the proof work. When  $p_{\mathrm{a}} = 1$  (no partial participation), the update rules from (9) reduce to (7) and (8).
The theoretical analysis of the new algorithm became more complicated: unlike (7) and (8), the control variables  $h_i^{t+1}$  and  $g_i^{t+1}$  in (9) (see also main Algorithm 1) are coupled by the randomness from the partial participation. Going deeper into details, for instance, one can compare Lemma I.2 from (Tyurin and Richtárik, 2023) and Lemma 5, which both bound  $\left\| g_i^{t+1} - h_i^{t+1} \right\|^2$ . The former lemma does not use the knowledge about the update rules of  $h_i^{t+1}$ , works with one expectation  $\mathbb{E}_{\mathcal{C}}[\cdot]$ , uses only (4), (15), and (16). The latter lemma additionally requires and uses the structure of the update rule of  $h_i^{t+1}$  (the structure is very important in the lemma since the control variables  $h_i^{t+1}$  and  $g_i^{t+1}$  are coupled), surgically copes with the expectations  $\mathbb{E}_{\mathcal{C}}[\cdot]$  and  $\mathbb{E}_{p_a}[\cdot]$  (for instance, it is not trivial in each order one should apply the expectations), and uses the sampling lemma (Lemma 1). The same reasoning applies to other parts of the analysis and the finite-sum setting: the generalization of the previous algorithm and the additional randomness from the partial participation required us to rethink the previous proofs.
At the first reading of the proofs, we suggest the reader follow the proof of Theorem 2 in the gradient setting (DASHA-PP), which takes a small part of the paper. Although the appendix seems to be dense and large, the size is justified by the fact that we consider four different sub-algorithms, DASHA-PP, DASHA-PP-PAGE, DASHA-PP-FINITE-MVR, and DASHA-PP-MVR, and also PL-condition (The theory is designed so that the proofs do not repeat steps of each other and use one framework).
6 Theorems
We now present the convergence rates theorems of DASHA-PP in different settings. We will compare the theorems with the results of the current state-of-the-art methods, MARINA and DASHA, that work
$^3$  If  $\mathcal{C}_i\in \mathbb{U}(\omega)$  , then  $\mathcal{C}_i^{p_{\mathrm{a}}}\in \mathbb{U}(\omega +1 / p_{\mathrm{a}} - 1)$
in the full participation setting. Suppose that MARINA or DASHA converges to  $\varepsilon$ -solution after  $T$  communication rounds. Then, ideally, we would expect the convergence of the new algorithms to  $\varepsilon$ -solution after up to  $T / p_{\mathrm{a}}$  communication rounds due to the partial participation constraints<sup>4</sup>. The detailed analysis of the algorithms under Polyak-Lojasiewicz condition we provide in Section F. Let us define  $\Delta_0 \coloneqq f(x^0) - f^*$ .
6.1 Gradient Setting
Theorem 2. Suppose that Assumptions 1, 2, 3, 7 and 8 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,  $b = \frac{p_{\mathrm{a}}}{2 - p_{\mathrm{a}}}$
$$
\gamma \leq \left(L + \left[ \frac {4 8 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {1 6}{n p _ {\mathrm {a}} ^ {2}} \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \right] ^ {1 / 2} \widehat {L}\right) ^ {- 1},
$$
and  $g_{i}^{0} = h_{i}^{0} = \nabla f_{i}(x^{0})$  for all  $i\in [n]$  in Algorithm 1 (DASHA-PP), then  $\operatorname{E}\left[\left\| \nabla f(\widehat{x}^T)\right\| ^2\right]\leq \frac{2\Delta_0}{\gamma T}$ .
Let us recall the convergence rate of MARINA or DASHA, the number of communication rounds to get  $\varepsilon$ -solution equals  $\mathcal{O}\left(\frac{\Delta_0}{\varepsilon}\left[L + \frac{\omega}{\sqrt{n}}\widehat{L}\right]\right)$ , while the rate of DASHA-PP equals  $\mathcal{O}\left(\frac{\Delta_0}{\varepsilon}\left[L + \frac{\omega + 1}{p_{\mathrm{a}}\sqrt{n}}\widehat{L}\right]\right)$ . Up to Lipschitz constants factors, we get the degeneration up to  $1 / p_{\mathrm{a}}$  factor due to the partial participation. This is the expected result since each worker sends useful information only with the probability  $p_{\mathrm{a}}$ .
6.2 Finite-Sum Setting
Theorem 3. Suppose that Assumptions 1, 2, 3, 4, 7, and 8 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,  $b = \frac{p_{\mathrm{page}} p_{\mathrm{a}}}{2 - p_{\mathrm{a}}}$ , probability  $p_{\mathrm{page}} \in (0, 1]$
$$
\gamma \leq \left(L + \left[ \frac {4 8 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {(1 - p _ {p a g e}) L _ {\max} ^ {2}}{B}\right) + \frac {1 6}{n p _ {\mathrm {a}} ^ {2} p _ {p a g e}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {p a g e}) L _ {\max} ^ {2}}{B}\right) \right] ^ {1 / 2}\right) ^ {- 1}
$$
and  $g_{i}^{0} = h_{i}^{0} = \nabla f_{i}(x^{0})$  for all  $i\in [n]$  in Algorithm 1 (DASHA-PP-PAGE) then  $\operatorname{E}\left[\left\| \nabla f(\widehat{x}^T)\right\| ^2\right]\leq \frac{2\Delta_0}{\gamma T}$ .
We now choose  $p_{\mathrm{page}}$  to balance heavy full gradient and light mini-batch calculations. Let us define  $\mathbb{1}_{p_{\mathrm{a}}} := \sqrt{1 - \frac{p_{\mathrm{aa}}}{p_{\mathrm{a}}}} \in [0,1]$ . Note that if  $p_{\mathrm{a}} = 1$  then  $p_{\mathrm{aa}} = 1$  and  $\mathbb{1}_{p_{\mathrm{a}}} = 0$ .
Corollary 1. Let the assumptions from Theorem 3 hold and  $p_{page} = \frac{B}{(m + B)}$ . Then DASHA-PP-PAGE needs
$$
T := \mathcal {O} \left(\frac {\Delta_ {0}}{\varepsilon} \left[ L + \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\max }}{\sqrt {B}}\right) + \frac {1}{p _ {\mathrm {a}}} \sqrt {\frac {m}{n}} \left(\frac {\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L}}{\sqrt {B}} + \frac {L _ {\max }}{B}\right) \right]\right) \tag {10}
$$
communication rounds to get an  $\varepsilon$ -solution and the expected number of gradient calculations per node equals  $\mathcal{O}(m + BT)$ .
The convergence rate the rate of the current state-of-the-art method DASHA-PAGE without partial participation equals  $\mathcal{O}\left(\frac{\Delta_0}{\varepsilon}\left[L + \frac{\omega}{\sqrt{n}}\left(\widehat{L} +\frac{L_{\max}}{\sqrt{B}}\right) + \sqrt{\frac{m}{n}}\frac{L_{\max}}{B}\right]\right)$ . Let us closer compare it with (10). As expected, we see that the second term w.r.t.  $\omega$  degenerates up to  $1 / p_{\mathrm{a}}$ . Surprisingly, the third term w.r.t.  $\sqrt{m / n}$  can degenerate up to  $\sqrt{B} /p_{\mathrm{a}}$  when  $\widehat{L}\approx L_{\mathrm{max}}$ . Hence, in order to keep degeneration up to  $1 / p_{\mathrm{a}}$ , one should take the batch size  $B = \mathcal{O}\left(L_{\max}^{2} / \widehat{L}^{2}\right)$ . This interesting effect we analyze separately in Section C. The fact that the degeneration is up to  $1 / p_{\mathrm{a}}$  we check numerically in Section A.
In the following corollary, we consider Rand  $K$  compressors (see Definition 5) and show that with the particular choice of parameters, up to the Lipschitz constants factors, DASHA-PP-PAGE gets the
4We check this numerically in Section A.
5The choice of the compressor is driven by simplicity, and the following analysis can be used for other unbiased compressors.
optimal oracle complexity and SOTA communication complexity. Indeed, comparing the following result with (Tyurin and Richtárik, 2023, Corollary 6.6), one can see that we get the degeneration up to  $1 / p_{\mathrm{a}}$  factor, which is expected in the partial participation setting. Note that the complexities improve with the number of workers  $n$ .
Corollary 2. Suppose that assumptions of Corollary 1 hold,  $B \leq \min \left\{\frac{1}{p_{\mathrm{a}}}\sqrt{\frac{m}{n}},\frac{L_{\max}^{2}}{\mathbb{1}_{P_{\mathrm{a}}}^{2}\hat{L}^{2}}\right\}^{6}$ , and we use the unbiased compressor RandK with  $K = \Theta \left(\sqrt[3]{d / \sqrt{m}}\right)$ . Then the communication complexity of Algorithm 1 is  $\mathcal{O}\left(d + \frac{L_{\max}\Delta_0d}{p_{\mathrm{a}}\varepsilon\sqrt{n}}\right)$ , and the expected number of gradient calculations per node equals  $\mathcal{O}\left(m + \frac{L_{\max}\Delta_0\sqrt{m}}{p_{\mathrm{a}}\varepsilon\sqrt{n}}\right)$ .
The convergence rate of DASHA-PP-FINITE-MVR is provided in Section E.5.
6.3 Stochastic Setting
We define  $h^t \coloneqq \frac{1}{n} \sum_{i=1}^{n} h_i^t$ .
Theorem 4. Suppose that Assumptions 1, 2, 3, 5, 6, 7 and 8 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,  $b \in \left(0, \frac{p_{\mathrm{a}}}{2 - p_{\mathrm{a}}} \right]$ ,  $\gamma \leq \left(L + \left[ \frac{48\omega(2\omega + 1)}{np_{\mathrm{a}}^2} \left( \widehat{L}^2 + \frac{(1 - b)^2 L_\sigma^2}{B} \right) + \frac{12}{np_{\mathrm{a}}b} \left( \left(1 - \frac{p_{\mathrm{aa}}}{p_{\mathrm{a}}} \right) \widehat{L}^2 + \frac{(1 - b)^2 L_\sigma^2}{B} \right) \right]^{1/2}\right)^{-1}$ , and  $g_i^0 = h_i^0$  for all  $i \in [n]$  in Algorithm 1 (DASHA-PP-MVR). Then
$$
\begin{array}{l} \mathrm {E} \left[ \left\| \nabla f (\widehat {x} ^ {T}) \right\| ^ {2} \right] \leq \frac {1}{T} \left[ \frac {2 \Delta_ {0}}{\gamma} + \frac {2}{b} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \left(\frac {3 2 b \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {4 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)}{n p _ {\mathrm {a}}}\right) \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2}\right) \right] \\ + \left(\frac {4 8 b ^ {2} \omega (2 \omega + 1)}{p _ {\mathrm {a}} ^ {2}} + \frac {1 2 b}{p _ {\mathrm {a}}}\right) \frac {\sigma^ {2}}{n B}. \\ \end{array}
$$
In the next corollary, we choose momentum  $b$  and initialize vectors  $h_i^0$  to get  $\varepsilon$ -solution.
Corollary 3. Suppose that assumptions from Theorem 4 hold, momentum  $b = \Theta \left( \min \left\{ \frac{p_{\mathrm{a}}}{\omega} \sqrt{\frac{n \varepsilon B}{\sigma^2}}, \frac{p_{\mathrm{a}} n \varepsilon B}{\sigma^2} \right\} \right)$ ,  $\frac{\sigma^2}{n \varepsilon B} \geq 1$ , and  $h_i^0 = \frac{1}{B_{\mathrm{init}}} \sum_{k=1}^{B_{\mathrm{init}}} \nabla f_i(x^0; \xi_{ik}^0)$  for all  $i \in [n]$ , and batch size  $B_{\mathrm{init}} = \Theta \left( \frac{\sqrt{p_{\mathrm{a}}} B}{b} \right)$ , then Algorithm 1 (DASHA-PP-MVR) needs
$$
T := \mathcal {O} \left(\frac {\Delta_ {0}}{\varepsilon} \left[ L + \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {\sigma}{p _ {\mathrm {a}} \sqrt {\varepsilon} n} \left(\frac {\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L}}{\sqrt {B}} + \frac {L _ {\sigma}}{B}\right) \right] + \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon B}\right)
$$
communication rounds to get an  $\varepsilon$ -solution and the number of stochastic gradient calculations per node equals  $\mathcal{O}(B_{\mathrm{init}} + BT)$ .
The convergence rate of the DASHA-SYNC-MVR, the state-of-the-art method without partial participation, equals  $\mathcal{O}\left(\frac{\Delta_0}{\varepsilon}\left[L + \frac{\omega}{\sqrt{n}}\left(\widehat{L} +\frac{L_\sigma}{\sqrt{B}}\right) + \frac{\sigma}{\sqrt{\varepsilon n}}\frac{L_\sigma}{B}\right] + \frac{\sigma^2}{n\varepsilon B}\right)$ . Similar to Section 6.2, we see that in the regimes when  $\widehat{L}\approx L_{\sigma}$  the third term w.r.t.  $1 / \varepsilon^{3 / 2}$  can degenerate up to  $\sqrt{B} /p_{\mathrm{a}}$ . However, if we take  $B = \mathcal{O}\left(L_{\sigma}^{2} / \widehat{L}^{2}\right)$ , then the degeneration of the third term will be up to  $1 / p_{\mathrm{a}}$ . This effect we analyze in Section C. The fact that the degeneration is up to  $1 / p_{\mathrm{a}}$  we check numerically in Section A.
In the following corollary, we consider Rand  $K$  compressors (see Definition 5) and show that with the particular choice of parameters, up to the Lipschitz constants factors, DASHA-PP-MVR gets the optimal oracle complexity and SOTA communication complexity of DASHA-SYNC-MVR method. Indeed, comparing the following result with (Tyurin and Richtárik, 2023, Corollary 6.9), one can see that we get the degeneration up to  $1 / p_{\mathrm{a}}$  factor, which is expected in the partial participation setting. Note that the complexities improve with the number of workers  $n$ .
Corollary 4. Suppose that assumptions of Corollary 3 hold, batch size  $B \leq \min \left\{\frac{\sigma}{p_{\mathrm{a}}\sqrt{\varepsilon}n}, \frac{L_{\sigma}^2}{1_{p_{\mathrm{a}}}^2\hat{L}^2}\right\}$ , we take RandK compressors with  $K = \Theta\left(\frac{Bd\sqrt{\varepsilon n}}{\sigma}\right)$ . Then the communication complexity equals
$^6$  If  $\mathbb{1}_{p_{\mathrm{a}}} = 0$ , then  $\frac{L_{\sigma}^2}{\mathbb{1}_{p_{\mathrm{a}}}^2\hat{L}^2} = +\infty$
$\mathcal{O}\left(\frac{d\sigma}{\sqrt{p_{\mathrm{a}}}\sqrt{n\varepsilon}} + \frac{L_{\sigma}\Delta_0d}{p_{\mathrm{a}}\sqrt{n\varepsilon}}\right)$ , and the expected number of stochastic gradient calculations per node equals  $\mathcal{O}\left(\frac{\sigma^2}{\sqrt{p_{\mathrm{a}}}n\varepsilon} + \frac{L_{\sigma}\Delta_0\sigma}{p_{\mathrm{a}}\varepsilon^{3/2}n}\right)$ .
We are aware that the initial batch size  $B_{\mathrm{init}}$  can be suboptimal w.r.t.  $\omega$  in DASHA-PP-MVR in some regimes (see also (Tyurin and Richtárik, 2023)). This is a side effect of mixing the variance reduction of stochastic gradients and compression. However, Corollary 4 reveals that we can escape these regimes by choosing the parameter  $K$  of Rand  $K$  compressors in a particular way. To get the complete picture, we analyze the same phenomenon under PL condition (see Section F) and provide a new method DASHA-PP-SYNC-MVR (see Section G).
Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. (2017). QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NIPS), pages 1709-1720.
Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. (2019). Lower bounds for non-convex stochastic optimization. arXiv preprint arXiv:1912.02365.
Beznosikov, A., Horvath, S., Richtárik, P., and Safaryan, M. (2020). On biased compression for distributed learning. arXiv preprint arXiv:2002.12410.
Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. (2020). Lower bounds for finding stationary points i. Mathematical Programming, 184(1):71-120.
Chang, C.-C. and Lin, C.-J. (2011). LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):1-27.
Cutkosky, A. and Orabona, F. (2019). Momentum-based variance reduction in non-convex SGD. arXiv preprint arXiv:1905.10018.
Fang, C., Li, C. J., Lin, Z., and Zhang, T. (2018). SPIDER: Near-optimal non-convex optimization via stochastic path integrated differential estimator. In NeurIPS Information Processing Systems.
Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. (2016). Deep learning, volume 1. MIT Press.
Gorbunov, E., Burlachenko, K., Li, Z., and Richtárik, P. (2021). MARINA: Faster non-convex distributed learning with compression. In 38th International Conference on Machine Learning.
Horvath, S., Ho, C.-Y., Horvath, L., Sahu, A. N., Canini, M., and Richtárik, P. (2019a). Natural compression for distributed deep learning. arXiv preprint arXiv:1905.10988.
Horváth, S., Kovalev, D., Mishchenko, K., Stich, S., and Richtárik, P. (2019b). Stochastic distributed learning with gradient quantization and variance reduction. arXiv preprint arXiv:1904.05115.
Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448-456. PMLR.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Dennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. (2021). Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1-2):1-210.
Sapio, A., Canini, M., Ho, C.-Y., Nelson, J., Kalnis, P., Kim, C., Krishnamurthy, A., Moshref, M., Ports, D. R., and Richtárik, P. (2019). Scaling distributed machine learning with in-network aggregation. arXiv preprint arXiv:1903.06701.
Stich, S. U., Cordonnier, J.-B., and Jaggi, M. (2018). Sparsified SGD with memory. Advances in Neural Information Processing Systems, 31.
Suresh, A. T., Sun, Z., Ro, J. H., and Yu, F. (2022). Correlated quantization for distributed mean estimation and optimization. arXiv preprint arXiv:2203.04925.
Szlendak, R., Tyurin, A., and Richtárik, P. (2021). Permutation compressors for provably faster distributed nonconvex optimization. arXiv preprint arXiv:2110.03300.
Tyurin, A. and Richtárik, P. (2023). DASHA: Distributed nonconvex optimization with communication compression and optimal oracle complexity. International Conference on Learning Representations (ICLR).
Vogels, T., He, L., Koloskova, A., Karimireddy, S. P., Lin, T., Stich, S. U., and Jaggi, M. (2021). RelaySum for decentralized deep learning on heterogeneous data. Advances in Neural Information Processing Systems, 34.
Wangni, J., Wang, J., Liu, J., and Zhang, T. (2018). Gradient sparsification for communication-efficient distributed optimization. Advances in Neural Information Processing Systems, 31.
Xu, H., Ho, C.-Y., Abdelmoniem, A. M., Dutta, A., Bergou, E. H., Karatsenidis, K., Canini, M., and Kalnis, P. (2021). Grace: A compressed communication framework for distributed machine learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), pages 561-572. IEEE.
Zhao, H., Burlachenko, K., Li, Z., and Richtárik, P. (2021a). Faster rates for compressed federated learning with client-variance reduction. arXiv preprint arXiv:2112.13097.
Zhao, H., Li, Z., and Richtárik, P. (2021b). FedPAGE: A fast local stochastic gradient method for communication-efficient federated learning. arXiv preprint arXiv:2108.04755.
1 Introduction 1
2 Optimization Problem 1
2.1 Unbiased Compressors 2
2.2 Nodes Partial Participation Assumptions 3
3 Motivation and Related Work 4
4 Contributions 5
5 Algorithm Description and Main Challenges Towards Partial Participation 5
6 Theorems 7
6.1 Gradient Setting 8
6.2 Finite-Sum Setting 8
6.3 Stochastic Setting 9
A Numerical Verification of Theoretical Dependencies 15
A.1 Experiments in Partial Participation Setting 16
B Original DASHA and DASHA-MVR Methods 17
C Problem of Estimating the Mean in the Partial Participation Setting 18
D Auxiliary facts 19
D.1 Sampling Lemma 19
D.2 Compressors Facts 21
E Proofs of Theorems 21
E.1 Standard Lemmas in the Nonconvex Setting 22
E.2 Generic Lemmas 23
E.3 Proof for DASHA-PP 26
E.4 Proof for DASHA-PP-PAGE 30
E.5 Proof for DASHA-PP-FINITE-MVR 41
E.6 Proof for DASHA-PP-MVR 52
F Analysis of DASHA-PP under Polyak-Lojasiewicz Condition 65
F.1 Gradient Setting 65
F.2 Finite-Sum Setting 65
F.3 Stochastic Setting 66
F.4 Proofs of Theorems 67
F.4.1 Standard Lemma under Polyak-Lojasiewicz Condition 67
F.4.2 Generic Lemma 67
F.4.3 Proof for DASHA-PP under PL-condition 69
F.4.4 Proof for DASHA-PP-PAGE under PL-condition 71
F.4.5 Proof for DASHA-PP-MVR under PL-condition 76
G Description of DASHA-PP-SYNC-MVR 83
G.1 Proof for DASHA-PP-SYNC-MVR 85
A Numerical Verification of Theoretical Dependencies
(a) Finite-sum setting,  $K = 500$  in  $\operatorname{Rand}K$ .
(b) Stochastic setting,  $\sigma^2 / n_{\varepsilon B} = 10000$ , and  $K = 200$  in Rand  $K$ .
Figure 1: Classification task with the real-sim dataset.
Our main goal is to verify the dependences from the theory. We compare DASHA-PP with DASHA. Clearly, DASHA-PP can not generally perform better than DASHA. In different settings, we verify that the bigger  $p_{\mathrm{a}}$ , the closer DASHA-PP is to DASHA, i.e., DASHA-PP converges no slower than  $1 / p_{\mathrm{a}}$  times.
In all experiments, we take the real-sim dataset with dimension  $d = 20,958$  and the number of samples equals 72,309 from LIBSVM datasets (Chang and Lin, 2011) (under the 3-clause BSD license), and randomly split the dataset between  $n = 100$  nodes equally, ignoring residual samples. In the finite-sum setting, we solve a classification problem with functions
$$
f _ {i} (x) := \frac {1}{m} \sum_ {j = 1} ^ {m} \left(1 - \frac {1}{1 + \exp \left(y _ {i j} a _ {i j} ^ {\top} x\right)}\right) ^ {2}, \tag {11}
$$
where  $a_{ij} \in \mathbb{R}^d$  is the feature vector of a sample on the  $i^{\text{th}}$  node,  $y_{ij} \in \{-1, 1\}$  is the corresponding label, and  $m$  is the number of samples on the  $i^{\text{th}}$  node for all  $i \in [n]$ . In the stochastic setting, we consider functions
$$
f _ {i} \left(x _ {1}, x _ {2}\right) := \mathrm {E} _ {j \sim [ m ]} \left[ - \log \left(\frac {\exp \left(a _ {i j} ^ {\top} x _ {y _ {i j}}\right)}{\sum_ {y \in \{1 , 2 \}} \exp \left(a _ {i j} ^ {\top} x _ {y}\right)}\right) + \lambda \sum_ {y \in \{1, 2 \}} \sum_ {k = 1} ^ {d} \frac {\left\{x _ {y} \right\} _ {k} ^ {2}}{1 + \left\{x _ {y} \right\} _ {k} ^ {2}} \right], \tag {12}
$$
where  $x_{1}, x_{2} \in \mathbb{R}^{d}$ ,  $\{\cdot\}_{k}$  is an indexing operation,  $a_{ij} \in \mathbb{R}^{d}$  is a feature of a sample on the  $i^{\text{th}}$  node,  $y_{ij} \in \{1, 2\}$  is a corresponding label,  $m$  is the number of samples located on the  $i^{\text{th}}$  node, constant  $\lambda = 0.001$  for all  $i \in [n]$ .
The code was written in Python 3.6.8 using PyTorch 1.9 (Paszke et al., 2019). A distributed environment was emulated on a machine with Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 64 cores.
We use the standard setting in experiments<sup>7</sup> where all parameters except step sizes are taken as suggested in theory. Step sizes are finetuned from a set  $\{2^i \mid i \in [-10, 10]\}$ . We emulate the partial participation setting using  $s$ -nice sampling with the number of nodes  $n = 100$ . We consider the Rand $K$  compressor and take the batch size  $B = 1$ . We plot the relation between communication rounds and values of the norm of gradients at each communication round.
In the finite-sum (Figure 1a) and in the stochastic setting (Figure 1b), we see that the bigger probability  $p_{\mathrm{a}} = s / n$  to 1, the closer DASHA-PP to DASHA. Moreover, DASHA-PP with  $s = 10$  and  $s = 1$  converges approximately  $\times 10$  ( $= 1 / p_{\mathrm{a}}$ ) and  $\times 100$  ( $= 1 / p_{\mathrm{a}}$ ) times slower, accordingly. Our theory predicts such behavior.