SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3 provides a summary of the learning parameters, model, and dataset used in the experiments. The models were evaluated on samples from the test set that were unseen during training. The step size  $(\gamma)$  was tuned by running each baseline on a range of values and taking the setup with the best validation accuracy. The plots in Figure 6 show the CIFAR-10 convergence plots for EL-Oracle, 7-Regular static, and 8-U-EquiStatic (most prominent topologies) over different step sizes. The optimal step-sizes are  $\gamma = 0.1$  for Fully connected and 8-U-EquiStatic, and  $\gamma = 0.05$  for the remaining algorithms and topologies over CIFAR-10. For FEMNIST, the optimal step size is  $\gamma = 0.1$  for all algorithms and topologies. All experiments in Section 4 were conducted for a total of 3000 communication rounds and the experiments for homogeneous data distributions of CIFAR-10 (Figure 7) were conducted for a total of 1000 communication rounds. The experiments over the FEMNIST dataset were conducted for a total of 1440 communication rounds.
C.2 Computational resources
We perform experiments on 6 hyperthreading-enabled machines with dual Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz of 8 cores. All algorithms were implemented as part of Decentralizepy [11]. Each DL node was assigned 2 virtual cores. An experiment took at most 3 hours in wall-clock time for experiments in Section 4, 2 hours in wall-clock time for FEMNIST experiments, and 1 hour in wall-clock time for CIFAR-10 experiments in Appendix C, summing to a total of 1152 virtual CPU hours. Across all experiments that are presented in this article, including different seeds and learning rate tuning, the total virtual CPU time goes up to approximately 54700 hours.
C.3 Experiments with homogeneous data distributions
In this experiment, we focused on evaluating the performance of different communication topologies under IID data. The data was generated using a Dirichlet distribution with  $\alpha = 1.0$ . As a reference
Figure 7: Convergence and communication usage results for homogeneous data partitioning (Dirichlet distribution  $\alpha = 1.0$ ) of the CIFAR-10 dataset.
Figure 8: Communication rounds vs. top-1 test accuracy and (left) and communication volume per node vs. test accuracy (right) for the FEMNIST dataset.
baseline, we established a Fully connected topology. Additionally, we examined EL (EL-Oracle and EL-Local) in comparison to 7-Regular static and 8-U-EquiStatic.
We observed that the results were not particularly exciting (Figure 7), primarily due to the nature of the data distribution, which leaves limited room for improvement across the different topologies. In terms of accuracy, the Fully connected topology achieved roughly  $2\%$  better accuracy than the sparser topologies. EL-Oracle displayed a marginal advantage over the other approaches. However, the convergence behavior for EL-Local, 7-Regular static, and EL-Oracle was nearly overlapping, both in terms of the number of rounds and communication usage. On the other hand, the 8-U-EquiStatic approach lagged behind, exhibiting  $1\%$  lower accuracy.
Although significant improvements were not observed in IID data settings, it is worth noting that EL did not hinder convergence. In more realistic non-IID data distributions, EL demonstrated improved accuracy and reduced the network utilization, as shown in Figure 4, Section 4.
C.4 Performance of EL and baselines on FEMNIST
As an additional dataset, we employed FEMNIST [7] to evaluate EL against D-PSGD over other topologies. Figure 8 shows the convergence plots for the baselines against both EL-Oracle and EL-Local. We observe the same trend with Fully connected topology performing the best in terms of achieved top-1 accuracy. EL-Oracle outperforms EL-Local and both achieve higher top-1 accuracy compared to the static topologies 7-Regular static and 8-U-EquiStatic. The improvement in accuracy with EL (0.2% between EL-Oracle and 7-Regular static) does not look as evident as with the CIFAR-10 dataset because the room for improvement (difference between 7-Regular static and Fully connected) is quite small. This can be attributed to the homogeneity of the FEMNIST dataset and the complexity of the digit recognition task.
D Notes on Network Connectivity and EL Performance
The implementation of our decentralized scheme is built around the condition that all nodes can communicate with each other. This is similar to the assumptions of the EquiTopo topologies, a
competitor baseline [53]. However, we argue that the connectivity requirement is a bit more lenient, allowing our EL approach to function in a wide range of practical scenarios. In data center settings, it is common to train on clusters of highly interconnected GPUs, and all-to-all communication should be achievable in these settings. In edge settings, e.g., a network of mobile devices collaboratively training a model while keeping private datasets, the communication barrier might appear more substantial. Nonetheless, Internet networks are generally well-connected, which mitigates this concern. More importantly, from a practical point of view, even if pairwise communications encounter some barriers, the decentralized and randomized nature of EL-Oracle and EL-Local should still allow for effective model learning and convergence. The occasional lack of communication between specific nodes should not significantly impact the algorithm's performance, as model updates are still propagated through other communicating nodes, as long as the network is not partitioned.
EL is most useful in scenarios where every pair of nodes can communicate, but the total communication budget is limited. Our randomized communication scheme allows for efficient use of the limited resources while ensuring faster model convergence than conventional decentralized learning approaches.