SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conduct experiments on the robust linear regression problem. This problem is defined as  $\min_{w\in \mathbb{R}^d}\max_{\| r_i\| \leq D}\frac{1}{2N}\sum_{i = 1}^{N}(w^T (x_i + r_i) - y_i)^2 +\frac{\lambda}{2}\| w\|^2 -\frac{\beta}{2}\| r\|^2$  where  $w$  are model weights,  $\{x_{i},y_{i}\}_{i = 1}^{N}$  is the training dataset and  $r$  is artificially added noise. We use  $\ell_2$ -regularization on both  $w$  and  $r$ . We consider a network with  $n = 25$  devices and two types of datasets: synthetic and real. Synthetic data allows us to control the factor  $\delta$ , which measures statistical similarity of functions over different nodes. We take local datasets of size  $b = 100$  and generate the dataset  $\{\hat{x}_i,\hat{y}_i\}_{i = 1}^n$  at the server randomly, with each entry drawn from the Standard Gaussian distribution. The datasets at other devices  $(i = 2,\dots ,n)$  are obtained by perturbing  $\{\hat{x}_i,\hat{y}_i\}_{i = 1}^n$  with normal noise  $\xi_{i}$ , having zero mean and controlled variance. Real datasets are taken from LibSVM library [19]. We uniformly divide the whole dataset between devices. The solution  $z^{*}$  is found using a large number of iterations of Extra Gradient [50,42] method. Algorithms I and 2 are compared with methods from Table [Extra Gradient [50,42], Local SGDA [25], FedAvg-S [36], SCAFFOLD-S [36], SCAFFOLD-Catalyst-S [36], ESTVGM [11], SMMDS [16], MASHA [15], Optimistic MASHA [12], Accelerated Extra Gradient [51]) implemented in Python 3.7. All the methods are tuned as in the theory of the corresponding papers. The number of full operator transmissions by one of the arbitrary devices is used as the  $x$ -axis. The experiment results are shown in Figure I. It is evident that Algorithm I and 2 consistently outperform all other competing algorithms by a significant margin. Furthermore, as the value of  $\delta$  decreases, the superiority over Local SGDA, SCAFFOLD-S, SCAFFOLD-Catalyst-S, MASHA, and Optimistic MASHA becomes even more pronounced.
(a) synthetic, small  $\delta$
(b) synthetic, medium  $\delta$
(c) synthetic, large  $\delta$
(d) synthetic,  $\delta \sim L$
(e) real, a9a
(f) real, w7a
Figure 1: Comparison of state-of-the-art methods for distributed VI. The comparison is made on synthetic datasets with small, medium, and large  $\delta$ , as well as the real datasets a9a, w7a, w8a from LibSVM. The  $x$ -axis denotes the number of full operators transmitted by one of the devices.
(g) real, w8a
We presented two new algorithms for solving distributed VI and SPP: Algorithm 1 which combines compression, similarity, and local steps techniques; and Algorithm 2 which uses partial participation instead of compression. Both algorithms have the best communication complexity, outperforming existing methods in theory and practice. For future work we plan to i) derive new methods to stochastic settings (finite-sum or bounded variance setups); ii) obtain lower bounds on communication complexity with compression.
This research of A. Beznosikov has been supported by The Analytical Center for the Government of the Russian Federation (Agreement No. 70-2021-00143 dd. 01.11.2021, IGK 000000D730321P5Q0002).