SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. We assume that  $n$  is decomposed as  $n = n_1 \times \dots \times n_L$  with minimum  $L$  where  $n_l \in [k + 1]$  for all  $l \in [L]$ . Without loss of generality, we suppose  $n_1 \leq n_2 \leq \dots \leq n_L$ . Then, for any  $i \neq j$ , it holds that  $n_i \times n_j \geq k + 2$  because if  $n_i \times n_j \leq k + 1$  for some  $i$  and  $j$ , this contradicts the assumption that  $L$  is minimum.
When  $L$  is even, we have
$$
n = \left(n _ {1} \times n _ {2}\right) \times \dots \times \left(n _ {L - 1} \times n _ {L}\right) \geq \left(k + 2\right) ^ {\frac {L}{2}}.
$$
Then, we get  $L \leq 2\log_{k + 2}(n)$ .
Next, we discuss the case when  $L$  is odd. When  $L \geq 3$ ,  $n_L \geq \sqrt{k + 2}$  holds because  $n_{L - 2} \times n_{L - 1} \geq k + 2$ . Thus, we get
$$
n = \left(n _ {1} \times n _ {2}\right) \times \dots \times \left(n _ {L - 2} \times n _ {L - 1}\right) \times n _ {L} \geq \left(k + 2\right) ^ {\frac {L - 1}{2}} \times n _ {L} \geq \left(k + 2\right) ^ {\frac {L}{2}}.
$$
Then, we get  $L \leq 2\log_{k + 2}(n)$  when  $L \geq 3$ .
Thus, given the case when  $L = 1$ , the length of the  $k$ -PEER HYPER-HYPERCUBE GRAPH is less than or equal to  $\max \{1, 2\log_{k + 2}(n)\}$ .
Lemma 2 (Length of SIMPLE BASE- $(k + 1)$  GRAPH). For any number of nodes  $n \in \mathbb{N}$  and maximum degree  $k \in [n - 1]$ , the length of the SIMPLE BASE- $(k + 1)$  GRAPH is less than or equal to  $2\log_{k + 1}(n) + 2$ .
Proof. When all prime factors of  $n$  are less than or equal to  $k + 1$ , the SIMPLE BASE- $(k + 1)$  GRAPH is equivalent to the  $k$ -PEER HYPER-HYPERCUBE GRAPH and the statement holds from Lemma 1. In the following, we consider the case when there exists a prime factor of  $n$  that is larger than  $k + 1$ . Note that because when  $L = 1$  (i.e.,  $n = a_1 \times (k + 1)^{p_1}$ ), all prime factors of  $n$  are less than or equal to  $k + 1$ , we only need to consider the case when  $L \geq 2$ . We have the following inequality:
$$
\begin{array}{l} \log_ {k + 1} (n) = \log_ {k + 1} \left(a _ {1} (k + 1) ^ {p _ {1}} + \dots + a _ {L} (k + 1) ^ {p _ {L}}\right) \\ \geq p _ {1} + \log_ {k + 1} (a _ {1}) \\ \geq p _ {1}. \\ \end{array}
$$
Then, because  $|V_1| = a_1 \times (k + 1)^{p_1}$ , it holds that  $m_1 = |\mathcal{H}_k(V_1)| \leq 1 + p_1 \leq \log_{k + 1}(n) + 1$ . Similarly, it holds that  $|\mathcal{H}_k(V_{1,1})| = p_1 \leq \log_{k + 1}(n)$  because  $|V_{1,1}| = (k + 1)^{p_1}$ . In Alg. 2, the update rule  $b_1 \gets b_1 + 1$  in line 22 is executed for the first time when  $m = m_1 + 2$  because  $L \geq 2$ . Thus, the length of the SIMPLE BASE- $(k + 1)$  GRAPH is at most  $m_1 + |\mathcal{H}_k(V_{1,1})| + 1 \leq 2\log_{k + 1}(n) + 2$ . This concludes the statement.
Lemma 3 (Length of BASE-  $(k + 1)$  GRAPH). For any number of nodes  $n\in \mathbb{N}$  and maximum degree  $k\in [n - 1]$ , the length of the BASE-  $(k + 1)$  GRAPH is less than or equal to  $2\log_{k + 1}(n) + 2$
Proof. The statement follows immediately from Lemma 2 and line 12 in Alg. 3.  $\square$
E Convergence Rate of DSGD over Various Topologies
Table 2 lists the convergence rates of DSGD over various topologies. These convergence rates can be immediately obtained from Theorem 2 stated in Koloskova et al. [11] and consensus rate of the topology. As seen from Table 2, the BASE-2 GRAPH enables DSGD to converge faster than the ring and torus and as fast as the exponential graph for any number of nodes, although the maximum degree of the BASE-2 GRAPH is only one. Moreover, for any number of nodes, the BASE- $(k + 1)$  GRAPH with  $2 \leq k < \lceil \log_2(n) \rceil$  enables DSGD to converge faster than the exponential graph, even though the maximum degree of the BASE- $(k + 1)$  GRAPH remains to be less than that of the exponential graph.
Table 2: Convergence rates and maximum degrees of DSGD over various topologies.
Topology	Convergence Rate	Maximum Degree	#Nodes n
Ring [28]	O\(\left(\frac{\sigma^2}{n\epsilon^2}+\frac{\zeta n^2+\sigma n}{\epsilon^{3/2}}+\frac{n^2}{\epsilon}\right)\cdot LF_0\)	2	∀n ∈ N
Torus [28]	O\(\left(\frac{\sigma^2}{n\epsilon^2}+\frac{\zeta n+\sigma\sqrt{n}}{\epsilon^{3/2}}+\frac{n}{\epsilon}\right)\cdot LF_0\)	4	∀n ∈ N
Exp. [43]	O\(\left(\frac{\sigma^2}{n\epsilon^2}+\frac{\zeta\log_2(n)+\sigma\sqrt{\log_2(n)}}{\epsilon^{3/2}}+\frac{\log_2(n)}{\epsilon}\right)\cdot LF_0\)	\(\lceil\log_2(n)\rceil\)	∀n ∈ N
1-peer Exp. [43]	O\(\left(\frac{\sigma^2}{n\epsilon^2}+\frac{\zeta\log_2(n)+\sigma\sqrt{\log_2(n)}}{\epsilon^{3/2}}+\frac{\log_2(n)}{\epsilon}\right)\cdot LF_0\)	1	A power of 2
1-peer Hypercube [31]	O\(\left(\frac{\sigma^2}{n\epsilon^2}+\frac{\zeta\log_2(n)+\sigma\sqrt{\log_2(n)}}{\epsilon^{3/2}}+\frac{\log_2(n)}{\epsilon}\right)\cdot LF_0\)	1	A power of 2
Base-(k+1) Graph (ours)	O\(\left(\frac{\sigma^2}{n\epsilon^2}+\frac{\zeta\log_{k+1}(n)+\sigma\sqrt{\log_{k+1}(n)}}{\epsilon^{3/2}}+\frac{\log_{k+1}(n)}{\epsilon}\right)\cdot LF_0\)	k	∀n ∈ N
F.1 Comparison of Base-  $(k + 1)$  and Simple Base-  $(k + 1)$  Graphs
Fig. 20 shows the length of the SIMPLE BASE- $(k + 1)$  GRAPH and BASE- $(k + 1)$  GRAPH. The results indicate that for all  $k$ , the length of the BASE- $(k + 1)$  GRAPH is less than the length of the SIMPLE BASE- $(k + 1)$  GRAPH in many cases.
Figure 20: Comparison of the length of the SIMPLE BASE- $(k + 1)$  GRAPH and BASE- $(k + 1)$  GRAPH.
F.2 Consensus Rate
In Fig. 21, we demonstrate how consensus error decreases on various topologies when the number of nodes  $n$  is a power of 2. The results indicate that the BASE-2 GRAPH and 1-peer exponential graph can reach the exact consensus after the same finite number of iterations and reach the consensus faster than other topologies. Note that the BASE-2 GRAPH is equivalent to the 1-peer hypercube graph when  $n$  is a power of 2.
Figure 21: Comparison of consensus rates among different topologies when the number of nodes  $n$  is a power of 2. Because the BASE-\{3,5\} GRAPH are the same as the BASE-\{2,4\} GRAPH, respectively, when  $n$  is a power of 2, we omit the results of the BASE-\{3,5\} GRAPH.
F.3 Decentralized Learning
F.3.1 Comparison of Base-  $(k + 1)$  Graph and EquiStatic
In this section, we compared the BASE- $(k + 1)$  GRAPH with the  $\{\mathrm{U},\mathrm{D}\}$ -EquiStatic [33]. The  $\{\mathrm{U},$  D)-EquiStatic are dense variants of the 1-peer  $\{\mathrm{U},\mathrm{D}\}$ -EquiDyn, and their maximum degree can be set as hyperparameters. We evaluated the  $\{\mathrm{U},\mathrm{D}\}$ -EquiStatic varying their maximum degrees; the results are presented in Fig. 22. In both cases with  $\alpha = 10$  and  $\alpha = 0.1$ , the BASE-2 GRAPH can achieve comparable or higher final accuracy than all  $\{\mathrm{U},\mathrm{D}\}$ -EquiStatic, and the BASE- $\{3,4,5\}$  GRAPH outperforms all  $\{\mathrm{U},\mathrm{D}\}$ -EquiStatic. Thus, the BASE- $(k + 1)$  GRAPH is superior to the  $\{\mathrm{U},$  D)-EquiStatic from the perspective of achieving a balance between accuracy and communication efficiency.
Figure 22: Test accuracy  $(\%)$  of DSGD with CIFAR-10 and  $n = 25$ . The number in the bracket is the maximum degree of a topology.
F.3.2 Comparison with Various Number of Nodes
In this section, we evaluated the effectiveness of the BASE- $(k + 1)$  GRAPH when varying the number of nodes  $n$ . Fig. 24 presents the learning curves, and Fig. 23 shows how consensus error decreases when  $n$  is 21, 22, 23, 24, and 25. From Fig. 24, the BASE-2 GRAPH consistently outperforms the 1-peer exponential graph and can achieve a final accuracy comparable to that of the exponential graph. Furthermore, the BASE- $\{3,4,5\}$  GRAPH can consistently outperform the exponential graph, even though the maximum degree of the BASE- $\{3,4,5\}$  GRAPH is less than that of the exponential graph.
In Fig. 25 presents the learning curve for  $n = 16$ . When the number of nodes is a power of two, the 1-peer exponential graph is also finite-time convergence, and the 1-peer exponential graph and BASE-2 GRAPH achieve competitive accuracy.
Figure 23: Comparison of consensus rates among different topologies. The number in the bracket denotes the maximum degree of a topology. We omit the results of the BASE-5 GRAPH when  $n = 24$  because the BASE-5 GRAPH and BASE-4 GRAPH are equivalent when  $n = 24$ .
Figure 24: Test accuracy  $(\%)$  of DSGD with CIFAR-10 and  $\alpha = 0.1$ . The number in the bracket denotes the maximum degree of a topology. When  $n = 24$ , we omit the results of the BASE-5 GRAPH because the BASE-5 GRAPH and BASE-4 GRAPH are equivalent. When  $n = 25$ , we omit the results of the BASE-6 GRAPH because the BASE-6 GRAPH and BASE-5 GRAPH are equivalent.
Figure 25: Test accuracy  $(\%)$  of DSGD with CIFAR-10 and  $n = 16$ . The number in the bracket is the maximum degree of a topology. We omit the results of the BASE-3 GRAPH and BASE-5 GRAPH because these graphs are equivalent to the BASE-2 GRAPH and BASE-4 GRAPH, respectively.
G Results with Other Neural Network Architecture
In this section, we evaluate the effectiveness of the BASE- $(k + 1)$  GRAPH with other neural network architecture. Fig. 26 shows the accuracy when we use ResNet-18 [5]. The results are consistent with the results with VGG-11 shown in Sec. 6.
Figure 26: Test accuracy  $(\%)$  of DSGD with  $n = 25$ , CIFAR-10, and ResNet. The number in the bracket denotes the maximum degree of a topology.
H Hyperparameter Setting
Tables 3 and 4 list the detailed hyperparameter settings used in Secs. 6 and F.3. We ran all experiments on a server with eight Nvidia RTX 3090 GPUs.
Table 3: Hyperparameter settings for Fashion MNIST with LeNet.
Dataset	Fashion MNIST
Neural network architecture	LeNet [15] with group normalization [40]
Data augmentation	RandomCrop of PyTorch
Step size	Grid search over {0.1, 0.01, 0.001}.
Momentum	0.9
Batch size	32
Step size scheduler	Cosine decay
Step size warmup	10 epochs
The number of epochs	200
Table 4: Hyperparameter settings for CIFAR-  $\{ {10},{100}\}$  with  $\{ \mathrm{{VGG}} - {11},\operatorname{ResNet} - {18}\}$  .
Dataset	CIFAR-{10, 100}
Neural network architecture	{VGG-11 [32], ResNet-18 [5]} with group normalization [40]
Data augmentation	RandomCrop, RandomHorizontalFlip, RandomErasing of PyTorch
Step size	Grid search over {0.1, 0.01, 0.001}.
Momentum	0.9
Batch size	32
Step size scheduler	Cosine decay
Step size warmup	10 epochs
The number of epochs	500