SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To further evaluate the communication efficiency of SGDM/Adam, SoftSignSGD and Birder with high bandwidth connections, we implement experiments for training ResNet-50 and BERT-Base with distributed nodes connected with 200Gbps InfiniBand. All the experimental settings are the same as we perform experiments with Ethernet in Subsection 5.1, and the experimental results are listed in Table 3 and Table 4.
As shown in Table 3 and Table 4, compared with the baseline SGDM/Adam, Birder can still reach up to  $1.45 \times$  speedup for ResNet-50 on ILSVRC2012 and  $2.85 \times$  speedup for BERT-Base on SQuAD 1.1, although the speed advantage is not so obvious as that with lower-bandwidth Ethernet connections. An interesting phenomenon
(a) Iteration-wise, BERT-Base, batch size  $= 16 \times 64$
(b) Time-wise, BERT-Base, batch size  $= 16 \times 64$
Figure 8: Iteration-wise and time-wise convergence speed for pre-training BERT-Base with 16 samples per GPU with 64 GPUs.
Table 3: System throughput (samples/s) of SGDM, 1-bit Adam and Birder for training ResNet-50 on ILSVRC2012 with 10Gbps Ethernet and 200Gbps InfiniBand.
#GPUs	Optimizer	Ethernet (10Gbps)	InfiniBand (200Gbps)
Throughput (samples/s)	Speedup	Scale Efficiency	Throughput (samples/s)	Speedup	Scale Efficiency
8	SGDM	3693	1.00×	100%	3693	1.00×	100%
1-bit Adam	3243	0.83×	100%	3243	0.83×	100%
Birder	3462	0.94×	100%	3462	0.94×	100%
16	SGDM	2959	1.00×	40.1%	4673	1.00×	63.2%
1-bit Adam	4715	1.60×	72.7%	5708	1.22×	88.0%
Birder	6015	2.03×	86.9%	6784	1.45×	97.9%
32	SGDM	4270	1.00×	28.9%	9063	1.00×	61.3%
1-bit Adam	7268	1.70×	56.0%	10249	1.13×	79.0%
Birder	9416	2.21×	68.0%	12131	1.34×	87.6%
32	SGDM	6189	1.00×	20.9%	16608	1.00×	56.2%
1-bit Adam	5546	0.89×	21.3%	16920	1.02×	65.2%
Birder	15253	2.47×	55.1%	19956	1.21×	72.1%
Table 4: System throughput (samples/s) of BertAdam, 1-bit Adam and Birder for fine tuning BERT-Base on SQuAD 1.1 with 10Gbps Ethernet and 200Gbps InfiniBand.
#GPUs	Optimizer	Ethernet (10Gbps)	InfiniBand (200Gbps)
Throughput (samples/s)	Speedup	Scale Efficiency	Throughput (samples/s)	Speedup	Scale Efficiency
8	BertAdam	413	1.00×	100%	413	1.00×	100%
1-bit Adam	358	0.87×	100%	358	0.83×	100%
Birder	412	1.00×	100%	412	0.94×	100%
16	BertAdam	84	1.00×	10.1%	272	1.00×	32.9%
1-bit Adam	213	2.54×	29.7%	522	1.92×	72.9%
Birder	431	5.13×	52.3%	776	2.85×	94.1%
32	BertAdam	119	1.00×	7.20%	543	1.00×	32.8%
1-bit Adam	274	2.30×	19.1%	903	1.66×	63.1%
Birder	730	6.13×	44.2%	1365	2.51×	82.9%
32	BertAdam	158	1.00×	4.78%	998	1.00×	30.2%
1-bit Adam	252	1.59×	8.80%	1496	1.50×	52.2%
Birder	990	6.26×	30.0%	2299	2.30×	69.8%
is that the system throughput of Birder with 10Gbps Ethernet can match that of SGDM/Adam with 200Gbps InfiniBand.
The experimental results in Table 3 and Table 4 also show that as the number of GPUs is increasing, the scale efficiency of SGDM/Adam, SoftSignSGD and Birder becomes lower. The reason for this phenomenon can be summarized in the following. When the number of GPUs doubles, the number of communication trips also
multiplies. We take the communication scheme All-Reduce for example. If the number of GPUs is  $n$ , each GPU requires  $2(n - 1)$  trips across the network confections. When the number is non-trivial, the computation time of the communication primitives may exceed the time of the pure communication itself and dominate the overall communication time, since the total communication overhead does not change with the number of GPUs. Notably, All-reduce is more efficient than All-to-All which is the core of our Hierarchical-1-bit-AllReduce. Hence, as shown in Table 3 and Table 4, the scale efficiency of Birder decreases more quickly than SGDM/Adam with the number of GPUs growing.
E Discussion
The original paper on 1-bit Adam reports a significant speed advantage (up to  $3.8\times$ ) for 1-bit Adam compared to full-precision Adam, with the advantage becoming more prominent as the number of GPUs increases. However, in our experiments, we did not observe clear speed advantages for 1-bit Adam over the original Adam. In fact, when running on 64 GPUs, 1-bit Adam was not only slower than the original Adam, but its throughput rate was even lower than that on 32 GPUs. There are several reasons for this phenomenon. First, the speedup of 1-bit Adam is obtained by comparing the throughput of the compression phase with that of the warm-up phase. However, in our experiments, we evaluated the overall average throughput of both the warm-up phase and the compression phase for 1-bit Adam. Second, the baseline Adam did not run with system-level efficient DDP. Third, the authors of 1-bit Adam customized highly efficient communication primitives specifically for their optimizer, whereas we utilized off-the-shelf communication primitives in PyTorch for all the optimizers to ensure fairness.
As shown in Figure 4, as the number of GPUs increases, the communication time for Birder also grows superlinearly. One of the reasons for this is that the communication primitive All-to-All accounts for an increasing portion of the communication time. However, the native All-to-All in Step (iii) of the Hierarchical-1-bit-All-Reduce is not less efficient than the native All-Reduce. Therefore, we plan to further optimize the All-to-All and All-Gather primitives to accelerate Birder.
When training large-scale DNNs, the mixed-precision technique is commonly used to reduce memory consumption, allowing for larger model sizes. While optimizers still utilize full-precision states and computations, which typically contribute to  $33 - 75\%$  of the total memory footprint, Birder does not require full-precision states or computations. Moreover, due to the random quantization of updates to 1 or -1, Birder can leverage lower precision than FP16 gradients to estimate the update. Therefore, Birder shows promise for applications that focus on reducing memory usage, as highlighted in recent research on 8-bit optimizers via block-wise quantization (Tim Dettmers et al., ICLR 2022).