SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martin Takáč
Variational inequalities are a broad and flexible class of problems that includes minimization, saddle point, and fixed point problems as special cases. Therefore, variational inequalities are used in various applications ranging from equilibrium search to adversarial learning. With the increasing size of data and models, today's instances demand parallel and distributed computing for real-world machine learning problems, most of which can be represented as variational inequalities. Meanwhile, most distributed approaches have a significant bottleneck – the cost of communications. The three main techniques to reduce the total number of communication rounds and the cost of one such round are the similarity of local functions, compression of transmitted information, and local updates. In this paper, we combine all these approaches. Such a triple synergy did not exist before for variational inequalities and saddle problems, nor even for minimization problems. The methods presented in this paper have the best theoretical guarantees of communication complexity and are significantly ahead of other methods for distributed variational inequalities. The theoretical results are confirmed by adversarial learning experiments on synthetic and real datasets.
Variational Inequalities (VI) have a rather long history of research. Their first applications were the equilibrium problems in economics and game theory [72, 34, 29]. Variational inequalities were seen as a universal paradigm, involving other attractive problems, including minimization, Saddle Point Problems (SPP), fixed point problems, and others. Whereas research for minimization was actively developed separately, research for saddle point problems was often coupled with science's development around variational inequalities. This trend continues even now. This is partly related to the fact that the methods for minimization problems are not always suitable for SPP and VI. Moreover, from the theoretical point of view, these methods give relatively poor or even zero guarantees of convergence. Therefore a separate branch of algorithms has been developed for VI and SPP. Such methods include, e.g. Extra Gradient [50, 71, 42], Tseng's method [84] and many others [14]. Over time, VI found new applications, e.g. in signal processing [77]. In the last decade, they also attracted the interest of the machine learning community. Nowadays, VI and SPP are used in various aspects of machine learning. For example, there are many applications for the already classical learning tasks in supervised learning (with non-separable loss [40]; with non-separable regularizer [6]), unsupervised learning (discriminative clustering [89]; matrix factorization [7]), image denoising [28, 17], robust optimization [10]. It turns out that VI can be also linked with modern areas of learning science, including reinforcement learning [73, 39], adversarial training [62], and GANs [31, 23, 30, 67, 20, 57, 74] to name a few.
Machine learning's rapid development offers more possibilities for consumers, but poses challenges for engineers. To meet modern society's demands, the performance and generalization capacity of the model must be improved. This is currently done by increasing training data and model size [5].
Hardware acceleration and most importantly distributed algorithm design are essential for effective handling of these workloads [85]. The distributed nature of the problem occurs not only due to a desire to speed up the learning process via parallelization, but because of the severe external constraints, e.g. as in Federated Learning (FL) [49, 56]. FL is a pioneering area of distributed learning, where users' personal devices are computing machines. Private data transfer from local devices is prohibited due to privacy constraints, making distributed approaches the only practical option. Most distributed algorithms have communications as a bottleneck. Communication wastes time, slowing down training/optimization and preventing effective parallelization. Different approaches can reduce communication rounds and cost per round. The first popular approach is the compression of transmitted information [2]; it helps to increase the speed of the communication round without much loss in the quality of learning. The second approach is use of local steps [64]; by increasing the number of local computations, it is sometimes possible to
Table 1: Summary of the results on the number of transmitted information in different approaches to communication bottleneck for distributed VI/SPP.
Method	Approach	Communication complexity
Extra Gradient [50,42]	O(Lμlog1/ε)
Local SGDA [25]	local steps	O(L2/5n2/5μ2/5ε1/5 + Lζ/μ3/2√ε)
FedAvg-S [36]	local steps	O(L2/μ2log1/ε + Lζ/μ2√ε)
SCAFFOLD-S [36]	local steps	O(L2/μ2log1/ε)
SCAFFOLD-Catalyst-S [36]	local steps	O(Lμlog2/1/ε)
ESTVGM [11]	local steps	O(Lμlog1/ε + Lζ/μ2√ε)
SMMDS [16]	similarity local steps	O([1 + δ/μ] log1/ε)
MASHA [15]	compression	O(L/√nμlog1/ε)
Optimistic MASHA [12]	similarity compression	O((L/nμ + δ/√nμ) log1/ε)
Accelerated Extra Gradient [51]	similarity local steps	O([1 + δ/μ] log1/ε)
Algorithm 1 (this paper)	similarity local steps compression	O((1 + δ/√nμ) log1/ε)
Algorithm 2 (this paper)	similarity local steps partial participation	O((1 + δ/√nμ) log1/ε)
Lower bound [16]	Ω(1 + δ/μ) log1/ε) (1)
Lower bound (this paper)	similarity local steps compression	Ω(1 + δ/√nμ) log1/ε)
Lower bound (this paper)	similarity local steps partial participation	Ω(1 + δ/√nμ) log1/ε)
(1) lower bound is deterministic and does not take into account the possibility of compression and partial participation. Notation:  $\mu =$  constant of strong monotonicity of the operator  $F$  (strong convexity - strong concavity of the corresponding saddle point problem),  $L =$  Lipschitz constant of  $F$ ,  $\delta =$  similarity parameter (typically,  $\delta \ll L$ ),  $\zeta =$  heterogeneity parameter,  $n =$  number of devices,  $b =$  local data size,  $\varepsilon =$  precision of the solution.
significantly reduce the number of communication rounds. Finally, the third basic approach relies on data similarity [79]. Local steps are often employed to help with this approach. Data similarity allows us to assume local loss functions on devices are similar, meaning local computations are close and replaceable. Both approaches and combinations have been extensively researched for minimizations, but a synergy of all three pillars of distributed algorithms has not been presented for either minimization problems or VI. Therefore, the purpose of this paper is to answer the following questions:
Is it possible to combine the three main techniques for dealing with the comm. bottleneck into one algorithm? Does the new method have better convergence in both theory and practice?
Recent years have seen a surge of research into effective communication techniques for distributed algorithms. Much of this work focuses on minimization problems, but there are also methods for SPP and VI that address the communication bottleneck issue. We summarize these approaches in Table I.
- Local methods. The use of local steps is probably the most popular approach to the communication bottleneck. Methods with local updates are also widely used in general distributed and FL. The idea of distributed minimization using methods similar to Local SGD is not an achievement of recent years and comes back to [64, 65, 91]. The development of theoretical analysis and new modifications for the original methods can be traced in the following works [58, 80, 46, 45, 87, 86, 48]. With the appearance of [69], local methods for distributed minimization problems get a new boost. The first local methods for VI and SPP were simply adaptations of local algorithms for the minimization problem. In particular, the Local SGD/FedAvg method was adapted in [25, 36]. [36] also analysed the Scaffold method [45] for SPP. In [11], the Extra Gradient method, a key method for SPP and VI, was modified by the use of local steps.
- Compression. Methods with compression also have a good background. Here we can highlight
the basic methods that add compression to GD or SGD [2], and modifications of this approach using memory [68]. Also of interest are works investigating biased compression operators [13] and error-compensation techniques [3, 81, 76]. For VI and SPP, methods with different compressions were proposed [15]. These algorithms are based on the variance reduction method. See also [76] and [15] for a more detailed survey of compressed methods for distributed minimization problems.
- Similarity. This paper examines the similarity of Hessians in distributed minimization problems, building on established theory and practice (Assumption 4.3). The first method for distributed minimization problems with data similarity was DANE [79]. The lower bounds on communications under similarity assumption was discussed in [4]. The optimal method was obtained in [90] (for quadratic problems) and [51] (for general case). For details and a complete picture of the methods for the distributed minimization problems with similarity, see [51] and [35]. The SPP were also considered under similarity assumption. In [16], both the lower bounds and the optimal method were obtained. In the current work, we "break down" these lower estimates by using compression (see Table 1 with footnote 1). Our current study advances these concepts by integrating compression techniques, significantly outperforming previous estimates (see Table 1 footnote 1). Most related methods (for the distributed problem with similarity) incorporate local steps.
- Local methods with compression. The combination of local updates and compression for minimization problems has been presented in [9, 70] (note that the theoretical rates are not optimal and used restrictive assumptions). Using the same local approach as in [69], in papers [63, 21], authors combine it with compression technique and get good convergence rates. For VI and SPP, there are no methods with both local steps and compression.
- Compression and similarity. Recently, a new class of compressions called permutation compressors was introduced for the distributed minimization problem under the data similarity assumption [82]. However, due to not using local steps technique, the gain from combining these two approaches was slight. For VI the close results were presented in [12].
3 Our contributions
We answer in the affirmative to the questions posed at the end of Section 1. In particular, our contributions are as follows.
- Three Pillars Algorithm. We present a new method Three Pillars Algorithm (Algorithm I) that combines three approaches: compression, similarity, and local steps for the efficient solution of distributed VI and SPP.
- The best communication complexity. We analyze the convergence of the new distributed algorithm for smooth strongly monotone VI and strongly convex-strongly concave SPP under similarity assumption. As can be seen in Table I our algorithm has better communication complexity than all available competitors. Since  $\delta \leq L$ , it is easy to see that our algorithms always outperform all methods except Optimistic MASHA [12]. One can note that in the worst case with  $\delta \sim L$  we repeat the results of Optimistic MASHA. But if the data are uniformly distributed, then  $\delta \sim \frac{L}{\sqrt{b}}$  or even  $\frac{L}{b}$  where  $b$  is the size of the local sample. Assume that  $N = bn$  is the size of the whole data, then our methods outperform Optimistic MASHA when  $L > \delta \sqrt{n}$ , i.e. when  $N > n^2$ . Such a ratio is typical because already classic modern datasets contain hundreds of thousands [54, 53] or even millions [24] of samples, and the number of devices rarely exceeds 100.
- Main calculations on server. Many distributed approaches, such as one of the most popular from [66], use the power of all computing devices evenly. But in terms of FL, we prefer to have the main computational workload on the server, rather than on the local devices of the users [16, 51]. Our algorithm has this feature.
- Extension with partial participation. We present a modified version of our first method - Algorithm 2 (see Appendix A). Instead of compression, one device is selected to send an uncompressed message. This reduces the local complexity on devices compared to Algorithm 1, while keeping the overall communication complexity the same.
- Lower bounds. To demonstrate the optimality of our proposed approaches, we establish lower bounds for the communication complexities. These estimates enhance those from [16], which did not consider the potential for compression and partial participation. Our results are shown to be unsurpassable in both scenarios involving compression and partial participation (see Table 1).
- Extension with stochastic local computations. Motivated by applications where computation is expensive, we present a modification of Algorithm 1 that emphasizes the use of stochastic operators in local computations, termed Algorithm 3 (see Appendix A).
- Non-monotone analysis. Although we focus primarily on strongly monotone VI, many machine learning problems are non-monotone, and hence we extend the convergence analysis to this case. Our
algorithm outperforms competitors in terms of communication complexity for both cases and is the first to gain acceleration in the non-monotone case.
- Experimental confirmation. Experimental results on adversarial training of regression models confirm the theoretical conclusion that both of our algorithms outperform their competitors.
4 Problem and assumptions
We denote standard inner product of  $x,y\in \mathbb{R}^d$  as  $\langle x,y\rangle = \sum_{i = 1}^{d}x_{i}y_{i}$ , where  $x_{i}$  corresponds to the  $i$ -th component of  $x$  in the standard basis in  $\mathbb{R}^d$ . It induces  $\ell_2$ -norm in  $\mathbb{R}^d$  in the following way  $\| x\| = \sqrt{\langle x,x\rangle}$ . Operator  $\mathbb{E}[\cdot ]$  denotes full mathematical expectation and  $\mathbb{E}_{\xi}[\cdot ]$  introduces the conditional mathematical expectation on  $\xi$ .
Formally, the problem of finding a solution for the variation inequality can be stated as follows:
$$
\text {F i n d} z ^ {*} \in \mathcal {Z} \text {s u c h t h a t} \langle F (z ^ {*}), z - z ^ {*} \rangle \geq 0, \forall z \in \mathcal {Z}, \tag {1}
$$
where  $F: \mathcal{Z} \to \mathbb{R}^d$  is an operator, and  $\mathcal{Z} \subseteq \mathbb{R}^d$  is a convex set. We assume that the training data describing  $F$  is distributed across  $n$  devices/nodes:  $F(z) = \frac{1}{n}\sum_{i=1}^{n} F_i(z)$ , where  $F_n: \mathcal{Z} \to \mathbb{R}^d$  for all  $i \in [n] := \{1,2,\ldots,n\}$ . To emphasize the extensiveness of the formalism (II), we give some examples of variational inequalities.
- Minimization. Consider the minimization problem:
$$
\min  _ {z \in \mathcal {Z}} f (z). \tag {2}
$$
Suppose that  $F(z) = \nabla f(z)$ . Then, if  $f$  is convex, it can be proved that  $z^{*} \in \mathcal{Z}$  is a solution for (1) if and only if  $z^{*} \in \mathcal{Z}$  is a solution for (2).
- Saddle point problem. Consider the saddle point problem:
$$
\min  _ {x \in \mathcal {X}} \max  _ {y \in \mathcal {Y}} g (x, y). \tag {3}
$$
Suppose that  $F(z) = F(x,y) = [\nabla_x g(x,y), -\nabla_y g(x,y)]$  and  $\mathcal{Z} = \mathcal{X} \times \mathcal{Y}$  with  $\mathcal{X} \subseteq \mathbb{R}^{d_x}, \mathcal{Y} \subseteq \mathbb{R}^{d_y}$ . Then, if  $g$  is convex-concave, it can be proved that  $z^* \in \mathcal{Z}$  is a solution for (1) if and only if  $z^* \in \mathcal{Z}$  is a solution for (3).
- Fixed point problem. Consider the fixed point problem:
$$
\text {F i n d} z ^ {*} \in \mathbb {R} ^ {d} \text {s u c h t h a t} T \left(z ^ {*}\right) = z ^ {*}, \tag {4}
$$
where  $T: \mathbb{R}^d \to \mathbb{R}^d$  is an operator. With  $F(z) = z - T(z)$ , it can be proved that  $z^{*} \in \mathcal{Z} = \mathbb{R}^{d}$  is a solution for (1) if and only if  $F(z^{*}) = 0$ , i.e.  $z^{*} \in \mathbb{R}^{d}$  is a solution for (4).
In order to prove convergence results, the following assumptions are introduced on the problem (1).
Assumption 4.1 (Lipschitzness). Each operator  $F_{i}$  is  $L$ -Lipschitz continuous on  $\mathcal{Z}$ , i.e. for all  $u, v \in \mathcal{Z}$  we have  $\| F_{i}(u) - F_{i}(v) \| \leq L \| u - v \|$ .
For problems (2) and (3),  $L$ -Lipschitzness of the operator means that the functions  $f(z)$  and  $f(x, y)$  are  $L$ -smooth.
Assumption 4.2 (Strong monotonicity). The operator  $F$  is  $\mu$ -strongly monotone on  $\mathcal{Z}$ , i.e. for all  $u, v \in \mathcal{Z}$  we have  $\langle F(u) - F(v), u - v \rangle \geq \mu \| u - v \|^2$ . Each operator  $F_i$  is monotone on  $\mathcal{Z}$ , i.e. strongly monotone with  $\mu = 0$ .
For problems (2) and (3), strong monotonicity of  $F$  means strong convexity of  $f(z)$  and strong convexity-strong concavity of  $f(x, y)$ .
Assumption 4.3 ( $\delta$ -relatedness in mean). The operators  $\{F_i\}$  is  $\delta$ -related in mean on  $\mathcal{Z}$ . It means that for any  $j$  operators  $\{F_i - F_j\}$  is  $\delta$ -Lipschitz continuous in mean on  $\mathcal{Z}$ , i.e. for all  $u, v \in \mathcal{Z}$  we have  $\frac{1}{n} \sum_{i=1}^{n} \|F_i(u) - F_j(u) - F_i(v) + F_j(v)\|^2 \leq \delta^2 \|u - v\|^2$ .
While Assumptions 4.1 and 4.2 are basic and widely known, Assumption 4.3 requires further remarks. This assumption goes back to the conditions of data similarity (as discussed in Section 2). In more detail, we consider distributed minimization (2) and saddle point (3) problems  $f(z) = \frac{1}{n}\sum_{i=1}^{n}f_{i}(z)$ ,  $f(x,y) = \frac{1}{n}\sum_{i=1}^{n}f_{i}(x,y)$ , and assume that for minimization local and global hessian are  $\delta$ -similar [79, 90, 35, 51], i.e.,  $\| \nabla^2 f_j(z) - \nabla^2 f_i(z) \| \leq \delta$ , and for SPP, second derivatives differ by  $\delta$  [16, 51]  $\| \nabla_{xx}^2 f_j(x,y) - \nabla_{xx}^2 f_i(x,y) \| \leq \delta$ ,  $\| \nabla_{xy}^2 f_j(x,y) - \nabla_{xy}^2 f_i(x,y) \| \leq \delta$ ,  $\| \nabla_{yy}^2 f_j(x,y) - \nabla_{yy}^2 f_i(x,y) \| \leq \delta$ . In the worst case  $\delta \sim L$ , but often for practical machine learning problems  $\delta < L$ . In particular, if we assume that the data is uniformly distributed between devices, it can be proven (see Theorem 1.3 from [83], Theorem 2 from [35]) that  $\delta = \tilde{\mathcal{O}}(L/\sqrt{b})$  or even  $\delta = \tilde{\mathcal{O}}(L/b)$ , where  $b$  is the number of local data points on each of the devices.
Let us note some of the terms we use in the paper. By local steps/updates, we mean that some device  $i$ , using only local information about operator  $F_{i}$ , changes the local value of variable  $z$ . Local computations/calculations include all computations of  $F_{i}$  on a local device: local updates/steps may take place, or the value of the operator is accumulated and is sent to the other device without changes
Algorithm 1 Three Pillars Algorithm
Parameters: stepsizes  $\gamma$  and  $\eta$ , momentum  $\tau$ , probability  $p \in (0;1]$ , number of local steps  $H$ ; Initialization: Choose  $z^0 = m^0 = (x^0, y^0) \in \mathcal{Z}$ ;  
1: for  $k = 0,1,\ldots,K-1$  do  
2: Server takes  $u_0^k = z^k$ ;  
3: for  $t = 0,1,\ldots,H-1$  do  
4: Server computes  $u_{t+1/2}^k = \operatorname{proj}_{\mathcal{Z}}[u_t^k - \eta(F_1(u_t^k) - F_1(m^k) + F(m^k) + \frac{1}{\gamma}(u_t^k - z^k - \tau(m^k - z^k))]$ ;  
5: Server updates  $u_{t+1}^k = \operatorname{proj}_{\mathcal{Z}}[u_t^k - \eta(F_1(u_{t+1/2}^k) - F_1(m^k) + F(m^k) + \frac{1}{\gamma}(u_{t+1/2}^k - z^k - \tau(m^k - z^k)))]$ ;  
6: end for  
7: Server broadcasts  $u_H^k$  and  $F_1(u_H^k)$  to devices;  
8: Devices in parallel compute  $Q_i(F_i(m^k) - F_1(m^k) - F_i(u_H^k) + F_1(u_H^k))$  and send to server;  
9: Server updates  $z^{k+1} = \operatorname{proj}_{\mathcal{Z}}[u_H^k + \gamma \cdot \frac{1}{n} \sum_{i=1}^{n} Q_i(F_i(m^k) - F_1(m^k) - F_i(u_H^k) + F_1(u_H^k))]$ ;  
10: Server updates  $m^{k+1} = \left\{ \begin{array}{ll} z^k, & \text{with probability } p, \\ m^k, & \text{with probability } 1-p, \end{array} \right.$ ;  
11: if  $m^{k+1} = z^k$  then  
12: Server broadcasts  $m^{k+1}$  to devices;  
13: Devices in parallel compute  $F_i(m^k)$  and send to server;  
14: Server computes  $F(m^{k+1}) = \frac{1}{n} \sum_{i=1}^{n} F_i(m^{k+1})$ ;  
15: end if  
16: end for
in local variables  $z$ . The local device  $i$  complexity we mean the number of local computations of  $F_{i}$  during the entire operation of algorithms, and by the total local complexity – the total number of computations of  $F_{i}$  for all  $i$ . A communication round refers to establishing a connection between two devices and transmitting information in one or both directions. When multiple devices communicate in parallel—for instance, if all nodes send information to a central device—it is counted as a single communication round. Since we employ sparsifiers as compressors, we quantify the transmitted information by the number of coordinates sent. Communication complexity is the total volume of information exchanged between all devices during the execution of the algorithm. Communication time, on the other hand, is the duration spent on all such communication rounds.
5 Three pillars algorithm
5.1 The algorithm
We consider a centralized communication architecture where the main device is the server (node with number 1, which stores data associated with  $F_{1}$ ). All other devices can send information to the server and receive responses from it. We follow the standard assumption that forwarding from devices to the server takes considerably longer than broadcasting from the server to the devices (see Section 3.5 from [43] and Section M.3 from [68]), hence we consider one-side compression from devices to the server. Our approach can be generalized to two-sided compression.
We split the idea of our new method Three Pillars Algorithm (Algorithm II) into five parts.
- Local problem. The idea of the simplest local method is that the devices make a lot of independent gradient steps, using only their local operators/gradients and sometimes averaging the current value of the optimization variables [64, 66]. With this approach, between communications, the devices minimize their local functions and can move quite far away if the number of local steps is large [47]. Therefore, the estimates for Local SGDA, FedAvg-S, ESTVGM from Table depend on the data heterogeneity parameter. This problem can be solved by modifying original local operators/functions via regularization. For example, we can ask to locally minimize not the local  $f_{i}(z)$ , but  $f_{i}(z) + \lambda \| z - z_{s} \|^2$ , where  $x_{s}$  is the last point synchronized with all devices [45]. Statistical preconditioning, the popular trick for most methods under similarity conditions, is also a regularization of the local problem [79, 90]. It also can be considered as Mirror Descent with special Bregman divergence (see Section 1.1. of [35] for more details). The papers [51, 16] suggest looking at the stochastic preconditioning in distributed optimization from a different perspective. The authors propose to rewrite the original distributed problem as a composite problem of two terms:  $\frac{1}{n} \sum_{i=1}^{n} f_{i}(x) = r(x) + q(x) = [f_{1}(x)] + [\frac{1}{n} \sum_{i=1}^{n} f_{i}(x) - f_{1}(x)]$ . The proximal method works well for composite problems with proximal-friendly functions (e.g., simple  $\ell_{2}$  regularizer) where the calculation of the proximal operator is for free, but, in the general case, the proximal operator is calculated by an auxiliary method with another iteration loop inside the main algorithm. The key idea in [51, 16]