SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Using the example in Fig. 4a, we explain the BASE- $(k + 1)$  GRAPH in more detail. Let  $G^{(1)}, \dots, G^{(4)}$  denote the graphs in Fig. 4a from left to right, respectively. In step 1, we split  $V$  into  $V_{1} := \{1, 2, 3\}$  and  $V_{2} := \{4, 5, 6\}$ . In step 2, nodes in  $V_{1}$  and nodes in  $V_{2}$  have the same parameter respectively by
exchanging parameters on  $G^{(1)}, \dots, G^{(3)}$  because the subgraphs composed on  $V_{1}$  and  $V_{2}$  are same as the SIMPLE BASE- $(k + 1)$  GRAPH (see Fig. 14a). Then, we construct  $U_{1} := \{1, 4\}$ ,  $U_{2} := \{2, 5\}$ , and  $U_{3} := \{3, 6\}$ . Finally, in step 3, all nodes reach the exact consensus by exchanging parameters in  $G^{(4)}$ .
Fig. 5 and Sec. F.1 compare the BASE- $(k + 1)$  GRAPH with the SIMPLE BASE- $(k + 1)$  GRAPH, demonstrating that the length of the BASE- $(k + 1)$  GRAPH is less than that of the SIMPLE BASE- $(k + 1)$  GRAPH in many cases. Moreover, Theorem 1 show the upper bound of the length of the SIMPLE BASE- $(k + 1)$  GRAPH and BASE- $(k + 1)$  GRAPH. The proof is provided in Sec. D.
Theorem 1. For any number of nodes  $n \in \mathbb{N}$  and maximum degree  $k \in [n - 1]$ , the length of the SIMPLE BASE- $(k + 1)$  GRAPH and BASE- $(k + 1)$  GRAPH is less than or equal to  $2\log_{k + 1}(n) + 2$ .
Figure 5: Comparison of length.
Corollary 1. For any number of nodes  $n \in \mathbb{N}$  and maximum degree  $k \in [n - 1]$ , the SIMPLE BASE- $(k + 1)$  GRAPH and BASE- $(k + 1)$  GRAPH are  $\mathcal{O}(\log_{k + 1}(n))$ -finite time convergence.
Therefore, the BASE- $(k + 1)$  GRAPH is a powerful extension of the 1-peer exponential [43] and 1-peer hypercube graphs [31] because they are  $\mathcal{O}(\log_2(n))$ -finite time convergence only if  $n$  is a power of 2 and their maximum degree cannot be set to any number other than 1.
5 Decentralized SGD on Base- $(k + 1)$  Graph
In this section, we verify the effectiveness of the BASE- $(k + 1)$  GRAPH for decentralized learning, demonstrating that the BASE- $(k + 1)$  GRAPH can endow DSGD with both a faster convergence rate and fewer communication costs than the existing topologies, including the ring, torus, and exponential graph. We consider the following decentralized learning problem:
$$
\min  _ {\boldsymbol {x} \in \mathbb {R} ^ {d}} \left[ f (\boldsymbol {x}) := \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (\boldsymbol {x}) \right], f _ {i} (\boldsymbol {x}) := \mathbb {E} _ {\xi_ {i} \sim \mathcal {D} _ {i}} \left[ F _ {i} (\boldsymbol {x}; \xi_ {i}) \right],
$$
where  $n$  is the number of nodes,  $f_{i}$  is the loss function of node  $i$ ,  $\mathcal{D}_i$  is the data distribution held by node  $i$ ,  $F_{i}(\pmb{x};\xi_{i})$  is the loss of node  $i$  at data sample  $\xi_{i}$ , and  $\nabla F_{i}(\pmb{x};\xi_{i})$  denotes the stochastic gradient. Then, we assume that the following hold, which are commonly used for analyzing decentralized learning methods [18, 20, 23, 43].
Assumption 1. There exists  $f^{\star} > -\infty$  that satisfies  $f(\pmb{x}) \geq f^{\star}$  for any  $\pmb{x} \in \mathbb{R}^{d}$ .
Assumption 2.  $f_{i}$  is  $L$  -smooth for all  $i\in [n]$
Assumption 3. There exists  $\sigma^2$  that satisfies  $\mathbb{E}_{\xi_i\sim \mathcal{D}_i}\| \nabla F_i(\pmb {x};\xi_i) - \nabla f_i(\pmb {x})\| ^2\leq \sigma^2$  for all  $\pmb {x}\in \mathbb{R}^d$
Assumption 4. There exists  $\zeta^2$  that satisfies  $\frac{1}{n}\sum_{i=1}^{n}\|\nabla f_i(\pmb{x}) - \nabla f(\pmb{x})\|^2 \leq \zeta^2$  for all  $\pmb{x} \in \mathbb{R}^d$ .
We consider the case when DSGD [18], the most widely used decentralized learning method, is used as an optimization method. Let  $\mathbf{W}^{(1)},\dots ,\mathbf{W}^{(m)}$  be mixing matrices of the BASE-  $(k + 1)$  GRAPH. In DSGD on the BASE-  $(k + 1)$  GRAPH, node  $i$  updates its parameter  $x_{i}$  as follows:
$$
\boldsymbol {x} _ {i} ^ {(r + 1)} = \sum_ {j = 1} ^ {n} W _ {i j} ^ {(1 + \mathrm {m o d} (r, m))} \left(\boldsymbol {x} _ {j} ^ {(r)} - \eta \nabla F _ {j} \left(\boldsymbol {x} _ {j} ^ {(r)}; \xi_ {j} ^ {(r)}\right)\right), \tag {1}
$$
where  $\eta$  is the learning rate. In this case, thanks to the property of finite-time convergence, DSGD on the BASE- $(k + 1)$  GRAPH converges at the following convergence rate.
Theorem 2. Suppose that Assumptions 1-4 hold. Then, for any number of nodes  $n \in \mathbb{N}$  and maximum degree  $k \in [n - 1]$ , there exists  $\eta$  such that  $\bar{\pmb{x}} := \frac{1}{n} \sum_{i=1}^{n} \pmb{x}_i$  generated by Eq. (1) satisfies  $\frac{1}{R + 1} \sum_{r = 0}^{R} \mathbb{E} \left\| \nabla f(\bar{\pmb{x}}^{(r)}) \right\|^2 \leq \epsilon$  after
$$
R = \mathcal {O} \left(\frac {\sigma^ {2}}{n \epsilon^ {2}} + \frac {\zeta \log_ {k + 1} (n) + \sigma \sqrt {\log_ {k + 1} (n)}}{\epsilon^ {3 / 2}} + \frac {\log_ {k + 1} (n)}{\epsilon}\right) \cdot L F _ {0} \tag {2}
$$
iterations, where  $F_0 \coloneqq f(\bar{\mathbf{x}}^{(0)}) - f^{\star}$ .
The above theorem follows immediately from Theorem 2 stated in Koloskova et al. [11] and Corollary 1. The convergence rates of DSGD over commonly used topologies are summarized in Sec. E. From Theorem 2 and Sec. E, we can conclude that for any number of nodes  $n$ , the BASE-2 GRAPH enables DSGD to converge faster than the ring and torus and as fast as the exponential graph, although the maximum degree of the BASE-2 GRAPH is only one. Moreover, if we set the maximum degree  $k$  to the value between 2 to  $\lceil \log_2(n) \rceil$ , the BASE- $(k + 1)$  GRAPH enables DSGD to converge faster than the exponential graph, even though the maximum degree of the BASE- $(k + 1)$  GRAPH remains less than that of the exponential graph. It is worth noting that if we increase the maximum degree of the 1-peer exponential and 1-peer hypercube graphs (i.e.,  $k$ -peer exponential and  $k$ -peer hypercube graphs with  $k \geq 2$ ), these topologies cannot enable DSGD to converge faster than the exponential graph because these topologies are no longer finite-time convergence even when the number of nodes is a power of 2.
In this section, we validate the effectiveness of the BASE-  $(k + 1)$  GRAPH. First, we experimentally verify that the BASE-  $(k + 1)$  GRAPH is finite-time convergence for any number of nodes in Sec. 6.1, and we verify the effectiveness of the BASE-  $(k + 1)$  GRAPH for decentralized learning in Sec. 6.2.
Figure 6: Comparison of consensus rates among various topologies. The number in the bracket indicates the maximum degree of a topology. Because the maximum degree of the exponential graph depends on  $n$ , the three numbers in the bracket indicate the maximum degree for each  $n$ .
(a)  $\alpha = 10$
Figure 7: Test accuracy  $(\%)$  of DSGD on various topologies with  $n = 25$ . The number in the bracket indicates the maximum degree of a topology. We also compared with dense variants of the 1-peer  $\{\mathrm{U}, \mathrm{D}\}$ -EquiDyn [33] in Sec. F.3.1, showing the superior performance of the BASE- $(k + 1)$  GRAPH.
(b)  $\alpha = 0.1$
6.1 Consensus Rate
Setup. Let  $x_{i} \in \mathbb{R}$  be the parameter that node  $i$  has, and let  $\bar{x} \coloneqq \frac{1}{n}\sum_{i=1}^{n}x_{i}$ . For each  $i$ , the initial value of  $x_{i}$  was drawn from Gaussian distribution with mean 0 and standard variance 1. Then, we evaluated how the consensus error  $\frac{1}{n}\sum_{i=1}^{n}(x_{i} - \bar{x})^{2}$  decreases when  $x_{i}$  is updated as  $x_{i} \gets \sum_{j=1}^{n}W_{ij}x_{j}$  where  $W$  is the mixing matrix associated with a given topology.
Results. Figs. 1 and 6 present how the consensus errors decrease on various topologies. The results indicate that the BASE- $(k + 1)$  GRAPH reaches the exact consensus after a finite number of iterations, while the other topologies only reach the consensus asymptotically. Moreover, as the maximum degree  $k$  increases, the BASE- $(k + 1)$  GRAPH reaches the exact consensus with fewer iterations. We also present the results when  $n$  is a power of 2 in Sec. F.2, demonstrating that the 1-peer exponential graph can reach the exact consensus as well as the BASE-2 GRAPH, but requires more iterations than the BASE-4 GRAPH.
6.2 Decentralized Learning
Next, we examine the effectiveness of the BASE- $(k + 1)$  GRAPH in decentralized learning.
Setup. We used three datasets, Fashion MNIST [41], CIFAR-\{10, 100\} [14], and used LeNet [15] for Fashion MNIST and VGG-11 [32] for CIFAR-\{10, 100\}. Additionally, we present the results using ResNet-18 [5] in Sec. G. The learning rate was tuned by the grid search and we used the cosine learning rate scheduler [22]. We distributed the training dataset to nodes by using Dirichlet distributions with hyperparameter  $\alpha$  [7], conducting experiments in both homogeneous and heterogeneous data distribution settings. As  $\alpha$  approaches zero, the data distributions held by each node become more heterogeneous. We repeated all experiments with three different seed values and reported their averages. See Sec. H for more detailed settings.
Results of DSGD on Various Topologies. We compared various topologies combined with the DSGD with momentum [4, 18], showing the results in Fig. 7. From Fig. 7, the accuracy differences among topologies are larger as the data distributions are more heterogeneous. From Fig. 7b, the BASE-\{2,3,4,5\} GRAPH reach high accuracy faster than the other topologies. Furthermore, comparing the final accuracy, the final accuracy of the BASE-2 GRAPH is comparable to or higher than that of the existing topologies, including the exponential graph.
Moreover, the final accuracy of the BASE-\{3,4,5\} GRAPH is higher than that of all existing topologies. From Fig. 7a, the accuracy differences among topologies become small when  $\alpha = 10$ ; however, the BASE-5 GRAPH still outperforms the other topologies. In Fig. 8, we present the results in cases other than  $n = 25$ , demonstrating that the BASE-2 GRAPH outperforms the 1-peer exponential graph and the BASE-\{3,4,5\} GRAPH can consistently outperform the exponential and 1-peer exponential graphs for all  $n$ . In Sec. F.3.2, we show the learning curves and the comparison of the consensus rate when  $n$  is 21, 22, 23, 24, and 25.
Figure 8: Test accuracy  $(\%)$  of DSGD with CIFAR-10 and  $\alpha = 0.1$
Results of  $D^2$  and QG-DSGDm on Various Topologies. The above results demonstrated that the BASE- $(k + 1)$  GRAPH outperforms the existing topologies, especially when the data distributions
are heterogeneous. Hence, we next compared the BASE-  $(k + 1)$  GRAPH with the existing topologies in the case where  $D^2$  [36] and QG-DSGDm [20], which are robust to data heterogeneity, are used as decentralized learning methods. From Fig.9, the BASE2 GRAPH can achieve comparable or higher accuracy than the 1-peer exponential graph, and the BASE-5 GRAPH consistently out
Figure 9: Test accuracy  $(\%)$  of  $D^2$  and QG-DSGDm with CIFAR-10,  $n = 25$ , and  $\alpha = 0.1$ .
performs the exponential graph. Thus, the BASE-  $(k + 1)$  GRAPH is useful not only for DSGD but also for  $D^2$  and QG-DSGDm and then enables these methods to achieve a reasonable balance between accuracy and communication efficiency.
In this study, we propose the BASE- $(k + 1)$  GRAPH, a novel topology with both a fast consensus rate and small maximum degree. Unlike the existing topologies, the BASE- $(k + 1)$  GRAPH is finite-time convergence for any number of nodes and maximum degree  $k$ . Thanks to this favorable property, the BASE- $(k + 1)$  GRAPH enables DSGD to obtain both a faster convergence rate and more communication efficiency than the existing topologies, including the ring, torus, and exponential graph. Through experiments, we compared the BASE- $(k + 1)$  GRAPH with various existing topologies, demonstrating that the BASE- $(k + 1)$  GRAPH enables various decentralized learning methods to more successfully reconcile accuracy and communication efficiency than the existing topologies.
Yuki Takezawa, Ryoma Sato, and Makoto Yamada were supported by JSPS KAKENHI Grant Number 23KJ1336, 21J22490, and MEXT KAKENHI Grant Number 20H04243, respectively.
[1] Assran, M., Loizou, N., Ballas, N., and Rabbat, M. (2019). Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning.
[2] Chen, Y., Yuan, K., Zhang, Y., Pan, P., Xu, Y., and Yin, W. (2021). Accelerating gossip sgd with periodic global averaging. In International Conference on Machine Learning.
[3] Cyffers, E., Even, M., Bellet, A., and Massoulie, L. (2022). Muffliato: Peer-to-peer privacy amplification for decentralized optimization and averaging. In Advances in Neural Information Processing Systems.
[4] Gao, H. and Huang, H. (2020). Periodic stochastic gradient descent with momentum for decentralized training. In arXiv.
[5] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition.
[6] Horváth, S. and Richtarik, P. (2021). A better alternative to error feedback for communication-efficient distributed learning. In International Conference on Learning Representations.
[7] Hsu, T.-M. H., Qi, and Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification. In arXiv.
[8] Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Dennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., D'Oliveira, R. G. L., Eichner, H., Rouayheb, S. E., Evans, D., Gardner, J., Garrett, Z., Gascon, A., Ghazi, B., Gibbons, P. B., Gruteser, M., Harchaoui, Z., He, C., He, L., Huo, Z., Hutchinson, B., Hsu, J., Jaggi, M., Javidi, T., Joshi, G., Khodak, M., Konecny, J., Korolova, A., Koushanfar, F., Koyejo, S., Lepoint, T., Liu, Y., Mittal, P., Mohri, M., Nock, R., Özgür, A., Pagh, R., Qi, H., Ramage, D., Raskar, R., Raykova, M., Song, D., Song, W., Stich, S. U., Sun, Z., Suresh, A. T., Tramér, F., Vepakomma, P., Wang, J., Xiong, L., Xu, Z., Yang, Q., Yu, F. X., Yu, H., and Zhao, S. (2021). Advances and open problems in federated learning. In Foundations and Trends in Machine Learning.
[9] Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. (2020). SCAF-FOLD: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning.
[10] Koloskova, A., Lin, T., Stich, S. U., and Jaggi, M. (2020a). Decentralized deep learning with arbitrary communication compression. In International Conference on Learning Representations.
A Detailed Explanation of  $k$ -peer Hyperhypercube Graph
In this section, we explain Alg. 1 in more detail. The  $k$ -PEER HYPER-HYPERCUBE GRAPH mainly consists of the following five steps.
Step 1. Decompose  $n$  as  $n = n_1 \times \dots \times n_L$  with minimum  $L$  such that  $n_l \in [k + 1]$  for all  $l \in [L]$ .
Step 2. If  $L = 1$ , we make all nodes obtain the average of parameters in  $V$  by using the complete graph. If  $L \geq 2$ , we split  $V$  into disjoint subsets  $V_{1}, \dots, V_{n_{L}}$  such that  $|V_{l}| = \frac{n}{n_{L}}$  for all  $l \in [n_{L}]$  and continue to step 3.
Step 3. For all  $l \in [n_L]$ , we make all nodes in  $V_{l}$  obtain the average of parameters in  $V_{l}$  by using the  $k$ -PEER HYPER-HYPERCUBE GRAPH  $\mathcal{H}_k(V_l)$ .
Step 4. We take  $n_L$  nodes from  $V_1, \dots, V_{n_L}$  respectively and construct a set  $U_1$ . Similarly, we construct  $U_2, \dots, U_{n_L}$  such that  $U_1, \dots, U_{n_L}$  are disjoint sets.
Step 5. For all  $l \in [n_L]$ , we make all nodes in  $U_{l}$  obtain the average of parameters in  $U_{l}$  by using the complete graph. Because the average of parameter  $U_{l}$  is equivalent to the average in  $V$  after step 4, all nodes reach the exact consensus.
When  $n \leq k + 1$ , the  $k$ -PEER HYPER-HYPERCUBE GRAPH becomes the complete graph because of step 2. When  $n > k + 1$ , we decompose  $n$  in step 1 and construct the  $k$ -PEER HYPER-HYPERCUBE GRAPH recursively in step 3. Thus, the  $k$ -PEER HYPER-HYPERCUBE GRAPH can make all nodes reach the exact consensus by the sequence of  $L$  graphs.
Using the example provided in Fig. 10, we explain the  $k$ -PEER HYPER-HYPERCUBE GRAPH in a more detailed manner. When  $n = 12$ , we decompose 12 as  $2 \times 2 \times 3$ . In step 2, we split  $V := \{1, \dots, 12\}$  into  $V_1 := \{1, \dots, 4\}$ ,  $V_2 := \{5, \dots, 8\}$ , and  $V_3 := \{9, \dots, 12\}$ . Step 3 corresponds to the first two graphs in Fig. 10b. As shown in Fig. 10a, the subgraphs consisting of  $V_1$ ,  $V_2$ , and  $V_3$  in the first two graphs in Fig. 10b are equivalent to the  $k$ -PEER HYPER-HYPERCUBE GRAPH with the number of nodes 4. Thus, all nodes reach the exact consensus by exchanging parameters in Fig. 10b.
(a)  $n = 4(= 2\times 2)$
(b)  $n = 12(= 2\times 2\times 3)$
Figure 10: Illustration of the 2-PEER HYPER-HYPERCUBE GRAPH. In Fig. 10a, all edge weights are  $\frac{1}{2}$ . In Fig. 10b, edge weights are  $\frac{1}{2}$  in the first two graphs and  $\frac{1}{3}$  in the last graph.
B Detailed Explanation of Simple Base-  $(k + 1)$  Graph with  $k\geq 2$
In Sec. 4.2, we explain Alg. 2 only in the case where maximum degree  $k$  is one. In this section, we explain the details of Alg. 2 in the case with  $k \geq 2$ .
The SIMPLE BASE- $(k + 1)$  GRAPH mainly consists of the following five steps.
Step 1. As in the base-  $(k + 1)$  number of  $n$ , we decompose  $n$  as  $n = a_{1}(k + 1)^{p_{1}} + \dots + a_{L}(k + 1)^{p_{L}}$  in line 1, and then split  $V$  into disjoint subsets  $V_{1}, \dots, V_{L}$  such that  $|V_{l}| = a_{l}(k + 1)^{p_{l}}$  for all  $l \in [L]$ .
Step 2. For all  $l \in [L]$ , we split  $V_{l}$  into disjoint subsets  $V_{l,1}, \dots, V_{l,a_{l}}$  such that  $|V_{l,a}| = (k + 1)^{p_{l}}$  for all  $a \in [a_{l}]$  in line 3.
Step 3. For all  $l \in [L]$ , we make all nodes in  $V_{l}$  obtain the average of parameters in  $V_{l}$  using the  $k$ -PEER HYPER-HYPERCUBE GRAPH  $\mathcal{H}_k(V_l)$  in line 11. Then, we initialize  $l'$  as one.
Step 4. Each node in  $V_{l' + 1} \cup \dots \cup V_L$  exchange parameters with  $a_{l'}$  nodes in  $V_{l'} (= V_{l',1} \cup \dots \cup V_{l',a_{l'}})$  such that the average in  $V_{l',a}$  becomes equivalent to the average in  $V$  for all  $a \in [a_{l'}]$ . We increase  $l'$  by one and repeat step 4 until  $l' = L$ . This procedure corresponds to line 15.
Step 5. For all  $l \in [L]$  and  $a \in [a_l]$ , we make all nodes in  $V_{l,a}$  obtain the average in  $V_{l,a}$  using the  $k$ -PEER HYPER-HYPERCUBE GRAPH  $\mathcal{H}_k(V_{l,a})$ . Since the average in  $V_{l,a}$  is equivalent to the average in  $V$  after step 4, all nodes reach the exact consensus. This procedure corresponds to line 25.
The major difference compared with the case where  $k = 1$  is step 4. In the case where  $k = 1$ , each node in  $V_{l' + 1} \cup \dots \cup V_L$  exchange parameters with one node in  $V_{l'}$  such that the average in  $V_{l'}$  becomes equivalent to that in  $V$ , while in the case where  $k \geq 2$ , each node in  $V_{l' + 1} \cup \dots \cup V_L$  exchange parameters such that the average in  $V_{l',a}$  becomes equivalent to that in  $V$  for all  $a \in [a_l]$ . Thanks to this step, we can make all nodes reach the exact consensus using  $k$ -PEER HYPER-HYPERCUBE GRAPH  $\mathcal{H}_k(V_{l,a})$  instead of  $\mathcal{H}_k(V_l)$  in step 5, and we can reduce the length of a graph sequence.
Using the example provided in Fig. 11, we explain Alg. 2 in a more detailed manner. Let  $G^{(1)}, \dots, G^{(4)}$  denote the graphs depicted in Fig. 11 from left to right, respectively. First, we split  $V := \{1, \dots, 7\}$  into  $V_1 := \{1, \dots, 6\}$  and  $V_2 := \{7\}$ , and then split  $V_1$  into  $V_{1,1} := \{1, 2, 3\}$  and  $V_{1,2} := \{4, 5, 6\}$ . In step 3, all nodes in  $V_1$  obtain the same parameter by exchanging parameters in  $G^{(1)}$  and  $G^{(2)}$ . In step 4, the average in  $V_{1,1}$ , that in  $V_{1,2}$ , and that in  $V_2$  become the same as the average in all nodes  $V$  by exchanging parameters in  $G^{(3)}$ . Thus, in step 5, all nodes reach the exact consensus by exchanging parameters in  $G^{(4)}$ .
Figure 11:  $k = 2, n = 7 (= 2 \times 3 + 1)$ . The value on the edge indicates the edge weight. For simplicity, we omit the edge value when it is  $\frac{1}{3}$ .
C Illustration of Topologies
C.1 Examples
Fig. 12 shows the examples of the SIMPLE BASE- $(k + 1)$  GRAPH. Using these examples, we explain how all nodes reach the exact consensus.
We explain the case depicted in Fig. 12a. Let  $G^{(1)}, G^{(2)}, G^{(3)}$  denote the graphs depicted in Fig. 12a from left to right, respectively. First, we split  $V := \{1, \dots, 5\}$  into  $V_1 := \{1, 2, 3\}$  and  $V_2 := \{4, 5\}$ , and then split  $V_2$  into  $V_{2,1} := \{4\}$  and  $V_{2,2} := \{5\}$ . After exchanging parameters in  $G^{(1)}$ , nodes in  $V_1$  and nodes in  $V_2$  have the same parameter respectively. Then, after exchanging parameters in  $G^{(2)}$ , the average in  $V_1$ , that in  $V_{2,1}$ , and that in  $V_{2,2}$  become the same as the average in  $V$ . Thus, by exchanging parameters in  $G^{(3)}$ , all nodes reach the exact consensus. Note that edge (4,5) in  $G^{(3)}$ , which is added in line 27 in Alg. 2, is not necessary for all nodes to reach the exact consensus because nodes 4 and 5 already have the same parameter after exchanging parameters in  $G^{(2)}$ ; however, it is effective in decentralized learning as we explained in Sec. 4.2.
(a)  $k = 2, n = 5(= 3 + 2)$
(b)  $k = 1, n = 7 (= 2^2 + 2 + 1)$
Figure 12: Illustration of the SIMPLE BASE- $(k + 1)$  GRAPH. The edge is colored in the same color as the line of Alg. 2 where the edge is added. The value on the edge indicates the edge weight. For simplicity, we omit the edge value when it is  $\frac{1}{3}$ .
C.2 Illustrative Comparison between Simple Base-  $(k + 1)$  and Base-  $(k + 1)$  Graphs
In this section, we provide an example of the SIMPLE BASE- $(k + 1)$  GRAPH, explaining the reason why the length of the BASE- $(k + 1)$  GRAPH is less than that of the SIMPLE BASE- $(k + 1)$  GRAPH.
Let  $G^{(1)}, \dots, G^{(5)}$  denote the graphs depicted in Fig. 13 from left to right, respectively.  $(G^{(1)}, G^{(2)}, G^{(3)}, G^{(4)}, G^{(5)})$  is finite-time convergence, but  $(G^{(1)}, G^{(2)}, G^{(3)}, G^{(5)})$  is also finite-time convergence because after exchanging parameters in  $G^{(3)}$ , nodes 3 and 4 already have the same parameters. Then, using the technique proposed in Sec. 4.3, we can remove such unnecessary graphs contained in the SIMPLE BASE- $(k + 1)$  GRAPH (see Fig. 4a). Consequently, the BASE- $(k + 1)$  GRAPH can make all nodes reach the exact consensus faster than the SIMPLE BASE- $(k + 1)$  GRAPH.
Figure 13: Illustration of the SIMPLE BASE-2 GRAPH with  $n = 6(= 2^2 + 2)$ . The edge is colored in the same color as the line of Alg. 2 where the edge is added.
C.3 Additional Examples
C.3.1 Simple Base-  $(k + 1)$  Graph
(a)  $n = 3$
(b)  $n = 4$
(c)  $n = 5$
(d)  $n = 6$
(e)  $n = 7$
(f)  $n = 8$
(g)  $n = 9$
(h)  $n = 10$
Figure 14: Illustration of the SIMPLE BASE-2 GRAPH with the various numbers of nodes.
(h)  $n = 10$
Figure 15: Illustration of the SIMPLE BASE-3 GRAPH with the various numbers of nodes.
C.3.2 Base-  $(k + 1)$  Graph
(h)  $n = 10$
Figure 16: Illustration of the BASE-2 GRAPH with the various numbers of nodes.
(h)  $n = 10$
Figure 17: Illustration of the BASE-3 GRAPH with the various numbers of nodes.
C.4 1-peer Hypercube Graph and 1-peer Exponential Graph
For completeness, we provide examples of the 1-peer hypercube [31] and 1-peer exponential graphs [43] in Figs. 19 and 18, respectively.
(a)  $n = 4$
(b)  $n = 8$
Figure 18: Illustration of the 1-peer hypercube graph. All edge weights are 0.5.
(b)  $n = 5$
(c)  $n = 6$
(d)  $n = 7$
(e)  $n = 8$
Figure 19: Illustration of the 1-peer exponential graph. All edge weights are 0.5.
Lemma 1 (Length of  $k$ -PEER HYPER-HYPERCUBE GRAPH). Suppose that all prime factors of the number of nodes  $n$  are less than or equal to  $k + 1$ . Then, for any number of nodes  $n \in \mathbb{N}$  and maximum degree  $k \in [n - 1]$ , the length of the  $k$ -PEER HYPER-HYPERCUBE GRAPH is less than or equal to  $\max \{1, 2\log_{k + 2}(n)\}$ .