SlowGuess/NeurIPS_2023_OCR huggingface dataset:
is to use  $r(x) = f_1(x)$  for the proximal operator. This implies the need of additional solving  $\arg \min_x\{f_1(x) + \lambda \| x - x_s\| \}$ , where  $x_{s}$  is some point. Note that no commutation is needed to calculate the proximal operator as to solution needs access only to  $f_{1}$  and can be solved locally. This reasoning shows that the idea with composite reformulation is also related to the regularization of the local problem. To extend the idea with composite reformulation into VI (see 2 for connections between min. prob. and VIs), we use the following proximal operator (see the loop in line 3 in Alg. 1):
Find  $\hat{u}^k\in \mathcal{Z}$  such that  $\langle G(\hat{u}^k),z - \hat{u}^k\rangle \geq 0,\forall z\in \mathcal{Z}$
with  $G(z) = F_{1}(z) + \frac{1}{\gamma} (z - v^{k})$  and  $v^{k} = z^{k} + \tau (m^{k} - z^{k}) - \gamma \cdot (F(m^{k}) - F_{1}(m^{k}))$ , where the essence of  $m^k$  will be explained later. An important issue to be addressed is the selection of the basic method that should be used for an inexact calculation of the proximal operator (we will discuss it in the next paragraphs). We also need to underline one more point. Our approach solves the local problem (proximal operator) only on the server that owns  $F_{1}$ . It is possible to solve different local problems on all devices in parallel as proposed in [79], but this leads to non-optimal convergence guarantees. The method from [79] was revised in [90], which proposed local steps on one device and improved convergence rates. In [45], an algorithm was constructed that uses local updates on all devices and achieves the same convergence results as in [90] (only in the quadratic case). If no improvement is gained from running local updates on all devices, why should we do it?
- Basic outer method. We need a basic method for solving the composite VI. Gradient Descent is one option for minimization problems, but VI and SPP have their own optimal methods. Therefore, the basic outer method for Algorithm 1 also needs to be specialized for VI. In [51, 16], the authors used Extra Gradient [50, 71, 42]. In our work, we selected Tseng's method [84] for this task.
- Local method. Since the subproblem in the inner loop from line 3 in Algorithm 1 is a VI with a strongly monotone and Lipschitz continuous operator, then it can be solved using the Extra Gradient method [50]. It is optimal for this class of problems.
- Use of compression. In order to use compression in the method, an additional technique is also required. Following [75, 32, 15], we take the idea of basing on the variance reduction technique from [1]. We introduce a reference sequence  $\{m^k\}_{k \geq 0}$  as in the classical variance reduction method [41]. At point  $m^k$ , we need to know the full values of operator  $F$ . When  $m^k$  is updated (line [10]), we transfer the full operators without compression (lines [12] and [13]). If probability  $p$  is small,  $m^k$  is rarely modified and hence condition [1] is satisfied with low probability. It turns out that in Algorithm [1] at each iteration all nodes send a compressed package and very rarely transmit a full operator. If  $p$  is chosen correctly, the effect of this is unnoticeable. But simply introducing an additional  $m^k$  sequence is not enough, we also need to add a negative momentum:  $\tau(m^k - z^k)$  (lines [4] and [5]). This is dictated by the nature of the variance reduction method for VI.
- Compression operators. In Algorithm [1](lines 8, 9) any unbiased compressor can be used, but to reveal the  $\delta$ -similarity ( $\delta$ -relatedness), we apply the so-called permutation compressors [82].
Definition 5.1 (Permutation compressors [82]). For different cases of  $n$  and  $d$ , we define
- for  $d \geq n$ . Assume that  $d \geq n$  and  $d = qn$ , where  $q \geq 1$  is an integer. Let  $\pi = (\pi_1, \dots, \pi_d)$  be a random permutation of  $\{1, \dots, d\}$ . Then for all  $u \in \mathbb{R}^d$  and each  $i \in \{1, 2, \dots, n\}$  we define  $Q_i(u) = n \cdot \sum_{j = q(i - 1) + 1}^{qi} u_{\pi_j} e_{\pi_j}$ .
- for  $d \leq n$ . Assume that  $n \geq d, n > 1$  and  $n = qd$ , where  $q \geq 1$  is an integer. Define the multiset  $S := \{1, \ldots, 1, 2, \ldots, 2, \ldots, d, \ldots, d\}$ , where each number occurs precisely  $q$  times. Let  $\pi = (\pi_1, \dots, \pi_n)$  be a random permutation of  $S$ . Then for all  $u \in \mathbb{R}^d$  and each  $i \in \{1, 2, \dots, n\}$  we define  $Q_i(u) = du_{\pi_i} e_{\pi_i}$ .
In practice, the case of  $d \geq n$  is more relevant. Permutation compressors require generating a new permutation of numbers from 1 to  $d$  each iteration. To ensure the same permutations are generated on all devices, a random seed can be defined in each iteration in a predetermined way (e.g. iteration counter). The permutation organizes how compression operators correspond. Take  $d = 12$  and  $n = 3$ : a permutation  $(5, 2, 10, 7, 4, 12, 1, 9, 3, 8, 11, 6)$  directs the first device to send coordinates  $5, 2, 10, 7$ ; the second  $4, 12, 1, 9$ ; the third  $3, 8, 11, 6$ . This system ensures non-overlapping data transmission, reducing the data sent per device by a factor of  $n$  and avoiding the zeroed "gradient" average that random selection risks. Permutation compressors thus correct redundancy and omission errors. They also capitalize on data similarity, ensuring the "gradient" at the server closely mirrors the true average. The original paper [82] recognized the benefits of this compressor with similar data but didn't fully leverage it due to its focus on non-convex settings without local steps.
- Summary. We use Tseng's method [84] for the composite VI, inaccurately computing the resolvent (proximal operator) with the Extra Gradient method [50]. Our convergence analysis is novel for Tseng's method and can be regarded as a byproduct and secondary contribution. We use a variance
reduction technique [1] for Tseng's method, remade into a compression technique in distributed settings [75, 32, 15], and choose a special compression for similarity setting [82].
5.2 Convergence
This section presents the convergence results of Algorithm  $\Pi$  and its analysis.
Theorem 5.2. Let  $\{z^k\}_{k > 0}$  denote the iterates of Algorithm  $\boxed{1}$  with compressors from Definition 5.1 for solving problem (1), which satisfies Assumptions 4.1, 4.2, 4.3. Then, if we choose the step sizes  $\gamma = \tilde{\mathcal{O}} (\min \{\frac{p}{\mu},\frac{\sqrt{p}}{\delta},\frac{H}{L}\})$ $\eta = \mathcal{O}((L + \frac{1}{\gamma})^{-1})$  and the momentum  $\tau = p$  then we have the convergence guarantee  $\mathbb{E}[\| z^K -z^*\|^2 ]\leq (1 - \frac{\gamma\mu}{2})^K\cdot 2\left\| z^0 -z^*\right\|^2$
In Algorithm 1, one mandatory communication round with compression occurs (lines 7 and 8) and possibly one more (without compression) with probability  $p$  (lines 12 and 13). Permutation compressor compress by a factor of  $n$ ; hence each iteration requires  $\mathcal{O}\left(\frac{1}{n} + p\right)$  data transfers from devices to the server on average: in line 8, we use compression and forward  $\frac{d}{n}$  coordinates from the devices to the server, in line 13, with probability  $p$  we transfer the full vector of  $d$  coordinates. If  $p$  is close to 1, Theorem 5.2 gives faster convergence, but more data transfer is needed. If  $p$  tends to 0, the transmitted information complexity per iteration decreases but the iterative convergence rate drops. The optimal choice of  $p$  is  $\frac{1}{n}$ , as stated in the corollary.
Corollary 5.3. Under the conditions of Theorem [5.2] the following number of the outer iterations is needed to achieve the accuracy  $\varepsilon$  (in terms of  $\mathbb{E}\left[\| z - z^{*}\|^{2}\right]\lesssim \varepsilon$ ) by Algorithm [7] with  $p = \frac{1}{n}$ :
$$
\mathcal {O} \left(\left[ n + \frac {\delta \sqrt {n}}{\mu} + \frac {L}{\mu H} \right] \log \frac {\| z ^ {0} - z ^ {*} \| ^ {2}}{\varepsilon}\right). \tag {5}
$$
Next, let us give a brief discussion of the previous estimate.
- Communications and local calculations. The iteration of Algorithm 1 has  $(1 + p)$  communication rounds and forwards  $\mathcal{O}\left(\frac{1}{n} + p\right)$  information on average. With  $p = \frac{1}{n}$ , the estimate in equation (5) is valid for both the number of iterations and communication rounds. To get an upper bound for the amount of information transmitted, divide (5) by  $n$ . The bound from Corollary 5.3 reflects the local complexity on all devices except the first (the server). To get an estimate for the first device, we need to multiply (5) by  $H$ .
- Optimal choice of  $H$ . It is clear from (5) that the number of local  $H$  steps can improve the number of communications, but there is a limit beyond which more iterations are not useful. Taking  $H = \lceil \frac{L}{\delta \sqrt{n}} \rceil$  is optimal. If there is no data similarity  $(\delta \sim L)$ , then one can choose  $H = 1$ , and Algorithm [1] communication complexity estimate becomes similar to those of MASHA and Optimistic MASHA. Choosing  $H = 1$  implies that local updates are meaningless when  $\delta \sim L$ . Papers [45, 88] on distributed minimization problems show that collecting large batches works just as well as methods that perform many local updates. Our paper considers deterministic methods, so we collect full batches and need local steps only within the presence of data similarity.
- "Break" lower bounds.  $H$  from the previous paragraph yields estimates of  $\tilde{\mathcal{O}}(n + \delta \sqrt{n} / \mu)$  and  $\tilde{\mathcal{O}}(1 + \delta / \sqrt{n}\mu)$  for the number of communications and transmitted information, respectively. These results are better than any existing methods and even superior to lower bounds for deterministic methods. Note that our method uses randomized compression and hence lower bounds for deterministic methods does not apply for Algorithm I (see Table I). To close this gap in Section 5.4 we provide new lower bound estimates that show the optimality of our approach.
- Variable server. In Algorithm 1, the server can be reselected deterministically or randomly at each outer iteration. Communication with the server can be decentralized using reduce and gather procedures [18], for this kind of procedures compression also greatly speeds up communication both theoretically and in practice [55]. Although the server can change, and decentralized communication networks can be used, it must still receive data from all devices. Asynchronous communication is not supported, the lack of communication with the device is also a problem for Algorithm 1. This is a potential problem that could slow down the method. This issue is partially solved by Algorithm 2 in the next section, which assumes communication with only 1 device per iteration.
5.3 Extension with partial participation
We also propose a modification of Algorithm  $\square$  with partial participation (see Algorithm 2 in Appendix A). The essence of this method is to replace lines 8, 9 of Algorithm  $\square$  with the following steps
8: Generate uniformly  $i_k$  from [n], device  $i_k$  computes  $F_{i_k}(u_H^k)$  and sends to server;
9: Server updates  $z^{k + 1} = \mathrm{proj}_{\mathcal{Z}}[u_H^k + \gamma \cdot (F_{i_k}(m^k) - F_1(m^k) - F_{i_k}(u_H^k) + F_1(u_H^k))]$ ;
The following theorem is valid for Algorithm 2.
Theorem 5.4. Let  $\{z^k\}_{k > 0}$  denote the iterates of Algorithm  $\boxed{2}$  for solving problem (1), which satisfies Assumptions  $\boxed{4.1}$ ,  $\boxed{4.2}$ ,  $\boxed{4.3}$ . Then, if we choose the step sizes  $\gamma = \tilde{\mathcal{O}}(\min\{\frac{p}{\mu}, \frac{\sqrt{p}}{\delta}, \frac{H}{L}\})$ ,  $\eta = \mathcal{O}((L + \frac{1}{\gamma})^{-1})$  and the momentum  $\tau = p$  then we have the convergence  $\mathbb{E}[\| z^K - z^*\|^2] \leq (1 - \frac{\gamma\mu}{2})^K \cdot 2 \| z^0 - z^*\|^2$ .
While in lines 8 and 9 of Algorithm 1 all devices compute the full local operators but forward only  $\mathcal{O}\left(\frac{1}{n}\right)$  information, in Algorithm 2 only one local device computes its operator and forwards this full operator. Hence, the total amount of transmitted information from all devices is equivalent for Algorithms 1 and 2. Therefore the optimal choice of  $p$  for Algorithm 2 is the same as for Algorithm 1.
Corollary 5.5. Under the conditions of Theorem 5.4,  $\mathcal{O}([n + \frac{\delta\sqrt{n}}{\mu} +\frac{L}{\mu H} ]\log \frac{\|z^0 - z^*\|^2}{\varepsilon})$  outer iterations are needed for the accuracy  $\varepsilon$  (in terms of  $\mathbb{E}\left[\| z - z^{*}\|^{2}\right]\lesssim \varepsilon$ ) by Algorithm 2 with  $p = \frac{1}{n}$ . Despite the fact that the estimates from Corollary 5.3 and 5.5 are the same, Algorithm 2 is better than Algorithm 1 in terms of local devices (but not the server) complexity. In line 8 of Algorithm 1 all devices collect full operator, while in Algorithm 2 only one node does it, then local complexity (not for first node) of Algorithm 2 is  $n$  times better. But while Algorithm 1 and Algorithm 2 are the same in terms of the number of transmitted information, Algorithm 2 is  $n$  times worse in terms of communication time. This is due to the fact that in line 8 of Algorithm 1 all devices in parallel send information using  $\mathcal{O}\left(\frac{1}{n}\right)$  units of time, while in Algorithm 2, one node need  $\mathcal{O}(1)$  units of time.
5.4 Lower bounds
To derive lower bounds, we define a "full package" as the amount of information transmitted in one complete communication round from all devices to the server without compression. We start from the lower bounds for algorithms with partial participation, as in the previous section.
Theorem 5.6. For any  $\delta, \mu > 0$  and  $n \in \mathbb{N}$  and  $K \in \mathbb{N}$ , there exists a distributed variational inequality (satisfying Assumptions 4.2 and 4.3) on  $\mathbb{R}^d$  (where  $d$  is sufficiently large) with  $z^* \neq 0$ , such that for any output  $\hat{z}$  of any distributed algorithm with partial participation after sending  $K$  full packages from the devices to the server, it holds that  $\mathbb{E}[\| \hat{z} - z^*\|^2]$  is  $\Omega(\exp(-16/(1 + \sqrt{\frac{\delta^2 n}{16\mu^2} + 1}) \cdot K)\| z^0 - z^*\|^2)$ .
The lower bounds confirm that results of the form  $(1 + \delta / (\sqrt{n}\mu))$  are optimal for distributed methods with partial participation; thus, our algorithm achieves this optimality. Lower bounds serve to present a "worst-case" distributed VI where any algorithm with partial participation cannot outperform the established bounds. For a formal classification of distributed algorithms with partial participation, an exemplification of a problematic case, and proofs of the lower bounds, see Appendix C.3. For the methods with compression, we can also show that our result is optimal. But here we have to restrict the class of compression operators and consider only random sparsifiers - random choice of coordinates for forwarding (Permutation compressors are suitable). See Appendix C.4 How to obtain lower guarantees for arbitrary compressors remains an open question.
5.5 Extension with stochastic local computations
In this section, we consider a stochastic formulation of  $\mathbb{1}$ , namely, we assume that each local term from  $\mathbb{1}$  also has the form of a finite sum:  $F_{i}(z) = \frac{1}{M}\sum_{j=1}^{M} F_{i,j}(z)$ . Although we can compute  $F_{i}$  locally on each  $i$ th device, we would like to avoid this due to computational cost. The stochastic finite sum setting is common for machine learning and refers to Monte Carlo approach and empirical risk minimization [78]. Moreover, such a setup is more than natural especially in the context of similarity. For example, if we initially have a large dataset  $N = nMb$  and distribute it to  $n$  devices, then each device has  $Mb$  samples:  $F = \frac{1}{n}\sum_{i=1}^{n} F_{i} = \frac{1}{n}\sum_{i=1}^{n}[\frac{1}{M}\sum_{j=1}^{M} F_{i,j}] = \frac{1}{n}\sum_{i=1}^{n}[\frac{1}{M}\sum_{j=1}^{M}(\frac{1}{b}\sum_{l=1}^{b} F_{i,j,l})]$ . The nature of similarity goes precisely that  $F_{i}$  have the form of a sub-sum taken from a whole large sum. As we noted in Section 4, the larger the sample size  $Mb$  for  $F_{i}$ , the smaller the similarity parameter  $\delta$ . Now in the stochastic case we consider  $F_{i,j}$  and they also have similarity, but with a larger  $\tilde{\delta}$ , since  $Mb$  is larger than  $b$ . To work with new setting we introduce two assumptions.
Assumption 5.7 (Lipschitzness in mean). Each operator  $F_{i}$  is  $\tilde{L}$ -Lipschitz continuous in mean on  $\mathcal{Z}$ , i.e. for all  $u, v \in \mathcal{Z}$  we have  $\frac{1}{M} \sum_{j=1}^{M} \| F_{i,j}(u) - F_{i,j}(v) \|^2 \leq \tilde{L}^2 \| u - v \|^2$ .
Assumption 5.8. The operators  $\{F_{i,j}\}$  is  $\tilde{\delta}$ -related in mean on  $\mathcal{Z}$ . For any  $i_1$  operators  $\{F_{i_1,j_1} - F_{i_2,j_2}\}$  is  $\tilde{\delta}$ -Lipschitz continuous in mean on  $\mathcal{Z}$ , i.e., for all  $u,v\in \mathcal{Z}$  we have  $\frac{1}{nM^2}\sum_{i_2 = 1}^{n}\sum_{j_1,j_2 = 1}^{M}\| F_{i_1,j_1}(u) - F_{i_2,j_2}(u) - F_{i_1,j_1}(v) + F_{i_2,j_2}(v)\|^2\leq \tilde{\delta}^2\| u - v\|^2$ .
The modified version of Algorithm  $\boxed{1}$  for the new setting looks as follows (see the full version in Appendix A - Algorithm 3):
2: Server takes  $u_0^k = w_0^k = z^k$ ;
4: Server computes
$$
u _ {t + 1 / 2} ^ {k} = \operatorname {p r o j} _ {\mathcal {Z}} \left[ \alpha u _ {t} ^ {k} + (1 - \alpha) w _ {t} ^ {k} - \eta \left(F _ {1} \left(w _ {t} ^ {k}\right) - F _ {1} \left(m ^ {k}\right) + F \left(m ^ {k}\right) + \frac {1}{\gamma} \left(w _ {t} ^ {k} - z ^ {k} - \tau \left(m ^ {k} - z ^ {k}\right)\right)\right) \right];
$$
5: Server updates
$$
\begin{array}{l} u _ {t + 1} ^ {k} = \operatorname {p r o j} _ {\mathcal {Z}} [ \alpha u _ {t} ^ {k} + (1 - \alpha) w _ {t} ^ {k} - \eta \left(F _ {1, j t} \left(u _ {t + 1 / 2} ^ {k}\right) - F _ {1, j t} \left(w _ {t} ^ {k}\right) + F _ {1} \left(w _ {t} ^ {k}\right) - F _ {1} \left(m ^ {k}\right) + F \left(m ^ {k}\right) \right. \\ + \frac {1}{\gamma} \left(u _ {t + 1 / 2} ^ {k} - z ^ {k} - \tau \left(m ^ {k} - z ^ {k}\right)\right) ], \text {w h e r e} j _ {t} \text {i s g e n e r a t e d u n i f o r m l y f r o m [ M ]}; \\ \end{array}
$$
Server updates  $w_{t+1}^{k} = \begin{cases} u_{t+1}^{k}, & \text{with probability } q, \\ w_{t}^{k}, & \text{with probability } 1 - q, \end{cases}$ ;
7: Server broadcasts  $u_{H}^{k}$  and  $F_{1,j_k^1}(u_H^k) - F_{1,j_k^1}(m^k)$  to devices, where  $j_{k}^{1}$  is generated uniformly from  $[M]$ ;
8: Devices in parallel compute  $Q_{i}(F_{i,j_{k}^{i}}(m^{k}) - F_{1,j_{k}^{1}}(m^{k}) - F_{i,j_{k}^{i}}(u_{H}^{k}) + F_{1,j_{k}^{1}}(u_{H}^{k}))$  and send to server, where  $j_{k}^{i}$  is generated uniformly from  $[M]$ ;
9: Server updates
$$
z ^ {k + 1} = \operatorname {p r o j} _ {\mathcal {Z}} \left[ u _ {H} ^ {k} + \gamma \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} Q _ {i} \left(F _ {i, j _ {k} ^ {i}} \left(m ^ {k}\right) - F _ {1, j _ {k} ^ {1}} \left(m ^ {k}\right) - F _ {i, j _ {k} ^ {i}} \left(u _ {H} ^ {k}\right) + F _ {1, j _ {k} ^ {1}} \left(u _ {H} ^ {k}\right)\right) \right];
$$
To deal with stochasticity we make two modifications in Algorithm 1: in lines 2-5 we switch the local algorithm from classical deterministic Extra Gradient to stochastic Extra Gradient from [1] (Algorithm 1), in lines 7-9 we use stochastic operators for all  $\{F_i\}$  instead of deterministic ones. The next theorem and corollary provides the convergence.
Theorem 5.9. Let  $\{z^k\}_{k\geq 0}$  denote the iterates of Algorithm  $\boxed{3}$  with compressors from Definition 5.1 for solving problem (1), which satisfies Assumptions 4.2, 5.7, 5.8. Then, if we choose the steplases  $\gamma = \tilde{\mathcal{O}} (\min \{\frac{p}{\mu},\frac{\sqrt{p}}{\delta},\frac{H}{\sqrt{M}\tilde{L}}\})$ $\eta = \mathcal{O}(M^{-1 / 2}(\tilde{L} +\frac{1}{\gamma})^{-1})$  , the momentums  $\tau = p$ $\alpha = 1 - q$  and the probability  $q = \frac{1}{M}$  then we have the convergence guarantee  $\mathbb{E}[\| z^K -z^*\|^2 ]\leq (1 - \frac{\gamma\mu}{2})^K$ $2\| z^{0} - z^{*}\|^{2}$
Corollary 5.10. Under the conditions of Theorem 5.9,  $\mathcal{O}([n + \frac{\delta\sqrt{n}}{\mu} +\frac{\sqrt{M}\tilde{L}}{\mu H} ]\log \frac{\|z^0 - z^*\|^2}{\varepsilon})$  outer iterations are needed for the accuracy  $\varepsilon$  (in terms of  $\mathbb{E}\left[\| z - z^{*}\|^{2}\right]\lesssim \varepsilon$ ) by Alg.3 with  $p = \frac{1}{n}$ .
5.6 Non-monotone case
In this section, we abandon the monotonicity assumption, but introduce the following assumption.
Assumption 5.11 (Non-monotonicity). The operator  $F$  is non-monotone (minty), if and only if there exists  $z^{*} \in \mathbb{R}^{d}$  such that for all  $z \in \mathbb{R}^{d}$  we have  $\langle F(z), z - z^{*} \rangle \geq 0$ .
This assumption is called the minty or variational stability condition. It is not a general non-monotonicity, but is already associated in the community with non-monotonicity [22, 38, 67, 61, 44, 37, 26], particularly with the setup, which is somewhat appropriate for GANS [59, 60, 27, 8].
Theorem 5.12. Let  $\{z^k\}_{k\geq 0}$  denote the iterates of Algorithm  $\boxed{1}$  for solving problem (1) with  $\mathcal{Z} = \mathbb{R}^d$ , which satisfies Assumptions 4.1, 5.11, 4.3. Then, if we additionally assume that  $\| z^k\| ,\| u_H^k\| ,\| z^*\| \leq \Omega$  and choose the step sizes  $\gamma = \frac{\sqrt{p}}{6\delta}$ ,  $\eta = \frac{1}{2(L + 1 / \gamma)}$ , the momentum  $\tau = p\leq \frac{1}{2}$ . then  $\frac{1}{K}\sum_{k = 0}^{K - 1}\mathbb{E}[\| F(u_H^k)\| ^2 ]\lesssim \frac{\delta^2\Omega^2}{p}\big(\frac{1}{K} +\frac{1}{\sqrt{H}}\big(\frac{L\sqrt{p}}{\delta} +1\big) + \frac{1}{H}\big(\frac{L\sqrt{p}}{\delta} +1\big)^2\big)$ .
By substituting  $H \sim K^2\left(\frac{L\sqrt{p}}{\delta} + 1\right)^2$ , we can obtain that  $\frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\left[\|F(u_H^k)\|^2\right] \lesssim \frac{\delta^2\Omega^2}{pK}$ . The discussion of the optimal choice of the parameter  $p$ , in this case, is the same as that following Theorem 5.2. Therefore, the following corollary is true.
Corollary 5.13. Under the conditions of Theorem 5.12  $\mathcal{O}(\frac{n\delta^2\Omega^2}{\varepsilon^2})$  outer iterations is needed to achieve the accuracy  $\varepsilon$  (in terms of  $\mathbb{E}\left[\| F(z)\| ^2\right]\lesssim \varepsilon^2$ ) by Algorithm with  $p = \frac{1}{n}$ .
One can note that the number of information transmitted is  $n$  times less than the number of iterations in Corollary 5.13. Therefore, we have the communication complexity is equal to  $\mathcal{O}\left(\delta^2\Omega^2 /\varepsilon^2\right)$ . This is better than the estimate  $\mathcal{O}\left(L^{2}\Omega^{2} / \varepsilon^{2}\right)$  of a basic method that does not use local steps, similarity, and compression [33]. The problem is that our estimate of communication complexity does not
benefit from large  $n$ . In the strongly monotone case, however, we observed a more pleasant situation (see Table I). Unfortunately, this is a standard problem of the non-monotone case. For example, the methods from [15] with compression have the same complexity  $\mathcal{O}\left(L^{2}\Omega^{2} / \varepsilon^{2}\right)$  as the basic method without compression. The methods from the paper [11] converge only in the special case when all local operators are equal to each other. Other papers from Table I do not consider the non-monotone case or consider it under other assumptions, then their complexity results are hard to compare with ours. Finally, it turns out that our work is the first to get an acceleration in terms of communication complexity in the non-monotone case. On the negative side, we can note that the estimate on the number of local iterations  $H$  in Theorem 5.12 is rather strict and grows with the number of iterations  $K$ . The results for Algorithm 2 in the non-monotone case can be obtained in a similar way.