SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f _ {i} (x) := \left\{ \begin{array}{l l} \frac {\mu}{2} \| x \| ^ {2} + \frac {L - \mu}{4} ([ x ] _ {1} ^ {2} + \sum_ {1 \leq r \leq d / 2 - 1} ([ x ] _ {2 r} - [ x ] _ {2 r + 1}) ^ {2} + [ x ] _ {d} ^ {2} - 2 [ x ] _ {1}), & \text {i f} i \leq n / 2, \\ \frac {\mu}{2} \| x \| ^ {2} + \frac {L - \mu}{4} (\sum_ {1 \leq r \leq d / 2} ([ x ] _ {2 r - 1} - [ x ] _ {2 r}) ^ {2}), & \text {i f} i > n / 2, \end{array} \right.
$$
where  $[x]_l$  denotes the  $l$ -th entry of vector  $x \in \mathbb{R}^d$ . We set  $\mu = 1$ ,  $L = 10^4$ ,  $d = 20$  and number of nodes  $n = 400$ .
Compressors. We apply various compressors to the algorithms with communication compression through our experiments. In the constructed quadratic problem, we consider ADIANA algorithm with random- $s$  compressors (see Example 1 in Appendix A) in six different settings, i.e., three choices of  $s$  ( $s = 1,2,4$ ), with two different (shared or independent) randomness settings. In the least squares and logistic regression problems, we apply the independent random- $\lfloor d / 20 \rfloor$  compressor to ADIANA, CANITA and DIANA algorithm. In particular, we use the unscaled version of the independent random- $\lfloor d / 20 \rfloor$  compressor for EF21 to guarantee convergence, where the values of selected entries are transmitted directly to the server without being scaled by  $d / s$  times. In Appendix F.2, we further apply independent natural compression [20] and random quantization [3] with  $s = \lceil \sqrt{d} \rceil$  in the above algorithms.
Figure 3: Convergence results of various distributed algorithms on a synthetic least squares problem (left), logistic regression problems with dataset a9a (middle) and w8a (right). The  $y$ -axis represents  $f(\hat{x}) - f^{\star}$  and the  $x$ -axis indicates the total communicated bits sent by per worker. All compressors used are independent natural compression.
Total communicated bits. For non-compression algorithms and algorithms with a fixed-length compressor, such as random-  $s$  and natural compression, the total communication bits can be calculated using the following formula: total communication bits = number of iterations  $\times$  communication rounds per iteration  $\times$  communicated bits per round. Among the algorithms we compare, ADIANA and CANITA communicate twice per iteration, while the other algorithms communicate only once. The communicated bits per round for non-compression algorithms amount to 64d for  $d$  float64 entries. In the case of the random-  $s$  compressor, the communicated bits per communication are calculated as  $64s + \lceil \log_2\binom{d}{s}\rceil$ . Similarly, for natural compression, the communicated bits per round are fixed at  $12d$ , with 1 sign bit and 11 exponential bits allocated for each entry. In the case of adaptive-length random quantization, the communication cost is evaluated using Elias integer encoding [16]. This cost is then averaged among  $n$  nodes, providing a more representative estimate.
Algorithm implementation. We implement ADIANA, CANITA, DIANA, EF21 algorithms following the formulation in Algorithm 1, [33], [26], and [49], respectively. We implement Nesterov's accelerated algorithm with the following recursions:
$$
\left\{ \begin{array}{l} y ^ {k} = (1 - \theta_ {t}) x ^ {k} + \theta_ {t} z ^ {k}, \\ x ^ {k + 1} = y ^ {k} - \eta_ {k} \nabla f (y ^ {k}), \\ z ^ {k + 1} = x ^ {k} + \frac {1}{\theta_ {k}} (x ^ {k + 1} - x ^ {k}). \end{array} \right.
$$
The value of  $\alpha$  in ADIANA, CANITA and DIANA are all set to  $1 / (1 + \omega)$ , and we set  $\gamma_{k},\beta$  of ADIANA as in Theorem 3. Other parameters are all selected through running Bayesian Optimization [45] for the first  $20\%$  iterations with 5 initial points and 20 trials. The exact value of the selected parameters are listed in Appendix F.3. Each curve (except for Nesterov's accelerated algorithm which does not involve randomness) is averaged through 20 trials, with the range of standard deviation depicted.
Computational resource. All experiments are run on an NVIDIA A100 server. Each trial consumes up to 10 minutes of running time.
F.2 Additional experiments
Additional compressors. In addition to the experiments in Sec. 6, we consider applying different compressors in the algorithms with communication compression. Fig. 3 and Fig. 4 show results of using natural compression and random quantization, respectively. These results are consistent with the results in Sec. 6.
CIFAR-10 dataset. We also consider binomial logistic regression with CIFAR-10 dataset, where labels of each datum are categorized by whether they equal to 3, i.e., the corresponding figures belong to the cat category. The full training set with 50000 images, are divided equally to  $n = 250$  nodes. The compressor choices follow the same strategies as in Appendix F.1, where dimension  $d = 3072$ . Fig. 5 compares convergence results between Nesterov method and ADIANA with different compressors. It can be observed that ADIANA equipped with more aggressive compressors, i.e., those with bigger  $\omega$ , benefits more from the compression, which is consistent with our theoretical results.
Figure 4: Convergence results of various distributed algorithms on a synthetic least squares problem (left), logistic regression problems with dataset a9a (middle) and w8a (right). The  $y$ -axis represents  $f(\hat{x}) - f^{\star}$  and the  $x$ -axis indicates the total communicated bits sent by per worker. All compressors used are independent random quantization.
Figure 5: Experimental results of logistic regression problem on the CIFAR-10 dataset. The objective function is constructed by relabeling the 10 classes into 2 classes, namely cat (corresponding to the original cat class) and non-cat (corresponding to the rest classes). ADIANA w. R.S. / R.Q. / N.C. represents ADIANA algorithm with random- $\lfloor d / 20 \rfloor$  compressor / random quantization compressor with  $s = \lceil \sqrt{d} \rceil$  / natural compression compressor, where  $d = 3072$  is the dimension of gradient vectors as well as the number of features in CIFAR-10 dataset. The experiments are conducted under the same setting as in the "Algorithm implementation" part in Appendix F.1.
F.3 Parameter values
In this subsection, we list all the parameter values that are selected by applying Bayesian Optimization. Table 2, 3, 4, 5, 6 list the parameters chosen in the least squares problem, logistic regression using a9a dataset, logistic regression using w8a dataset, the constructed problem, and logistic regression using CIFAR-10 dataset, respectively.
Table 2: Parameters for algorithms in the least squares problem. Notation R.S. stands for independent random sparsification, N.C. stands for independent natural compression, R.Q. stands for independent random quantization.
Algorithm	Parameters
Nesterov	η = 3.0 × 10-2, θ = 1.4 × 10-2.
ADIANA R.S.	η = 4.8 × 10-2, θ1 = 2.2 × 10-2, θ2 = 7.6 × 10-2, p = 4.1 × 10-2.
ADIANA N.C.	η = 3.9 × 10-2, θ1 = 1.0 × 10-2, θ2 = 2.9 × 10-1, p = 9.9 × 10-1.
ADIANA R.Q.	η = 6.5 × 10-2, θ1 = 1.4 × 10-2, θ2 = 2.7 × 10-1, p = 5.5 × 10-1.
DIANA R.S.	γ = 7.9 × 10-2.
DIANA N.C.	γ = 7.4 × 10-2.
DIANA R.Q.	γ = 7.6 × 10-2.
EF21 R.S.	γ = 6.2 × 10-2.
EF21 N.C.	γ = 6.8 × 10-2.
EF21 R.Q.	γ = 7.4 × 10-2.
Table 3: Parameters for algorithms in the logistic regression problem with a9a dataset. Notation  $k$  stands for the index of iteration. Other notations are as in Table 2.
Algorithm	Parameters
Nesterov	η = 9.4 × 10-1, θ = 1.7 × 10-1.
ADIANA R.S.	η = 2.1, θ1 = 1.3×101/k+5.2×102, θ2 = 2.1 × 10-1,p = 7.7 × 10-1.
ADIANA N.C.	η = 2.1, θ1 = 1.0/k+4.3, θ2 = 8.0 × 10-3,p = 8.0 × 10-1.
ADIANA R.Q.	η = 2.2, θ1 = 1.3/k+1.3, θ2 = 1.5 × 10-1,p = 8.5 × 10-1.
CANITA R.S.	η = min{k+2.1×102/2.1×102, 1.4}, θ = 2.0×101/k+2.3×102,p = 7.8 × 10-1.
CANITA N.C.	η = 1.2, θ = 2.1/k+1.2×101,p = 5.2 × 10-1.
CANITA R.Q.	η = 2.0, θ = 3.0/k+3.0,p = 7.2 × 10-1.
DIANA N.C.	γ = 2.6.
DIANA R.S.	γ = 9.4 × 10-1.
DIANA R.Q.	γ = 4.7 × 10-1
EF21 R.S.	γ = 1.3.
EF21 N.C.	γ = 1.6.
EF21 R.Q.	γ = 2.7.
Table 4: Parameters for algorithms in logistic regression with w8a dataset. Notations are as in Table 3.
Algorithm	Parameters
Nesterov	η = 1.5 × 10^1, θ = 9.4 × 10^-1.
ADIANA R.S.	η = min{k+4.1×10^2/1.2×10^2, 15}, θ1 = 8.8/k+4.8×10^2, θ2 = 2.4 × 10^-2, p = 3.6 × 10^-1.
ADIANA N.C.	η = 1.5 × 10^1, θ1 = 2.5/k+1.1×10^1, θ2 = 6.7 × 10^-1, p = 8.3 × 10^-1.
ADIANA R.Q.	η = 1.5 × 10^1, θ1 = 1.9/k+7.4, θ2 = 4.2 × 10^-1, p = 9.9 × 10^-1.
CANITA R.S.	η = min{k+2.0×10^2/2.2×10^2, 7.7}, θ = 1.1×10^1/k+2.3×10^2, p = 4.3 × 10^-1.
CANITA N.C.	η = min{k+1.1×10^1/2.7, 1.1 × 10^1}, θ = 5.4/k+7.4×10^1, p = 4.9 × 10^1.
CANITA R.Q.	η = min{k+1.6×10^1/7.3, 1.5 × 10^1}, θ = 2.2/k+2.2×10^1, p = 4.6 × 10^-1.
DIANA R.S.	γ = 1.5 × 10^1.
DIANA N.C.	γ = 1.6 × 10^1.
DIANA R.Q.	γ = 1.5 × 10^1.
EF21 R.S.	γ = 2.0 × 10^1.
EF21 N.C.	γ = 1.5 × 10^1.
EF21 R.Q.	γ = 1.5 × 10^1.
Table 5: Parameters for algorithms in the constructed problem. Notation i.d.rand-  $s$  denotes independent random-  $s$  compressor, s.d.rand-  $s$  denotes random-  $s$  compressor with shared randomness.
Algorithm	Parameters
Nesterov	η = 1.4 × 10-1, θ = 1.2 × 10-4.
ADIANA i.d.rand-1	η = 1.5 × 10-4, θ1 = 1.8 × 10-1, θ2 = 1.3 × 10-1, p = 1.5 × 10-1.
ADIANA i.d.rand-2	η = 1.5 × 10-4, θ1 = 1.5 × 10-4, θ2 = 5.0 × 10-2, p = 1.9 × 10-1.
ADIANA i.d.rand-4	η = 1.3 × 10-4, θ1 = 9.2 × 10-2, θ2 = 5.0 × 10-2, p = 2.3 × 10-1.
ADIANA s.d.rand-1	η = 1.4 × 10-6, θ1 = 2.0 × 10-2, θ2 = 1.6 × 10-1, p = 2.7 × 10-2.
ADIANA s.d.rand-2	η = 9.6 × 10-6, θ1 = 7.0 × 10-2, θ2 = 4.3 × 10-1, p = 1.8 × 10-1.
ADIANA s.d.rand-4	η = 1.6 × 10-5, θ1 = 6.0 × 10-2, θ2 = 2.1 × 10-1, p = 1.6 × 10-1.
Table 6: Parameters for algorithms in logistic regression with CIFAR-10 dataset. Notations are as in Table 4.
Algorithm	Parameters
Nesterov	η = 1.1 × 10-1, θ = 1.5 × 10-1.
ADIANA R.S.	η = 1.4 × 10-1, θ1 = 12/k+3.3×102, θ2 = 9.0 × 10-2, p = 4.3 × 10-1.
ADIANA N.C.	η = 1.4 × 10-1, θ1 = 1.0×10-2/k+7.0, θ2 = 7.0 × 10-1, p = 8.5 × 10-1.
ADIANA R.Q.	η = 1.2 × 101, θ1 = 8.2/k+59, θ2 = 8.0 × 10-1, p = 6.0 × 10-1.