SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanyang Peng $^{1*}$ , Shuang Qin $^{1*}$ , Yue Yu $^{1\dagger}$ , Jin Wang $^{1}$ , Hui Wang $^{1}$ , Ge Li $^{2\dagger}$
$^{1}$ Peng Cheng Laboratory, Shenzhen, China
$^{2}$ School of Electronic and Computer Engineering,
Shenzhen Graduate School, Peking University, Shenzhen, China
penghy@pcl.ac.cn, qinsh@pcl.ac.cn, yuy@pcl.ac.cn
wangj05@pcl.ac.cn, wangh06@pcl.ac.cn, geli@ece.pku.edu.cn
Various gradient compression algorithms have been proposed to alleviate the communication bottleneck in distributed learning, and they have demonstrated effectiveness in terms of high compression ratios and theoretical low communication complexity. However, when it comes to practically training modern deep neural networks (DNNs), these algorithms have yet to match the inference performance of uncompressed SGD-momentum (SGDM) and adaptive optimizers (e.g., Adam). More importantly, recent studies suggest that these algorithms actually offer no speed advantages over SGDM/Adam when used with common distributed DNN training frameworks (e.g., DistributedDataParallel (DDP)) in the typical settings, due to heavy compression/decompression computation or incompatibility with the efficient All-Reduce or the requirement of uncompressed warmup at the early stage. For these reasons, we propose a novel 1-bit adaptive optimizer, dubbed Binary randomization adaptive optimizer (Birder). The quantization of Birder can be easily and lightly computed, and it does not require warmup with its uncompressed version in the beginning. Also, we devise Hierarchical-1-bit-All-Reduce to further lower the communication volume. We theoretically prove that it promises the same convergence rate as the Adam. Extensive experiments, conducted on 8 to 64 GPUs (1 to 8 nodes) using DDP, demonstrate that Birder achieves comparable inference performance to uncompressed SGDM/Adam, with up to  $2.5 \times$  speedup for training ResNet-50 and  $6.3 \times$  speedup for training BERT-Base. Code is publicly available at https://openi.pcl.ac.cn/c2net_optim/Birder.
With the rapid development of computational power, "bigger" and "bigger" deep neural network (DNN) models are proposed for expect better performance, from the early classical models, such as AlexNet(61 million parameters) [15], and ResNet (ResNet-50: 20.5 million parameters) [12] to the current large language models (LLMs), such as BERT (BERT-Lagre: 340 million parameters) [10], and GPT (GPT-3: 176 billion parameters)[5]. Scalable parallelism across distributed computing workers for training these large-scale models becomes a necessity. During training, millions to billions of parameters need to be communicated among workers at each iteration, so distributed large-scale DNN training almost invariably suffers from the communication bottleneck.
To address the communication bottleneck, a wide variety of lossy gradient compression algorithms have been proposed to lower the communication overhead. The algorithms can be broadly divided
into three categories based on the compression techniques, including low-precision approximation (e.g., SignSGD[4], TernGrad [31], and QSGD [3], 1-bit Adam [28]), low-rank simplification (e.g., ATOMO[30], PowerSGD [29], and GradZip [9]), and sparsification (e.g., Random- $k$  [27], Top- $k$  [2], and MSTop- $k$  [24]). In addition to the specific compression techniques, some other works, such as Error Feedback(EF) [23][26], EF21 [21], DIANA [19] and MARINA [11], focus on changing the compression objects from the gradient to the gradient and delayed error summation, or the gradient differences to mitigate compressing errors and/or accelerate the convergence rate.
Gradient compression algorithms have demonstrated promising results with a high compression ratio and low oracle/communication complexity in theory. However, when practically training DNNs, they are still inferior to uncompressed SGDM/Adam in terms of inference performance. This is because, these gradient compression algorithms are SGD-type optimizers, which will be commonly reduced to vanilla SGD without momentum if compression is not employed. The performance for a compressed optimizer is commonly upper bounded by its uncompressed counterpart, while vanilla SGD is typically less effective than SGDM for training DNNs. Particularly, SGD-type optimizers are known to be substantially inferior to adaptive optimizers (e.g., Adam) for training Transformer-based networks [17] [34] [7], which have become predominant in the DNN community. Supporting empirical evidences for this phenomenon can be found in Section B in the Appendix. Furthermore, if we apply the techniques of gradient compression algorithms to compress and communicate the gradients, and subsequently utilize the compressed gradients to construct adaptive optimizers in the local nodes, the final performance will be degraded [28]. Therefore, designing native communication-compression adaptive optimizers is an underexplored problem that requires further research.
As for the system-level speed, recent studies ([33],[1]) pointed out, when distributedly training typical DNN models (e.g., ResNet-50 and BERT-Base) with off-the-shelf DistributedDataParallel (DDP) at typical bandwidths (e.g., 10Gbps), existing gradient compression algorithms are still slower than uncompressed SGDM/Adam. This is because, the compressor for these algorithms are either quantization or sparsification or low-rank simplification, which exhibit one or more weaknesses below. (i) Vector-wise quantization and low-rank simplification compressors are commonly computationally heavy, and their time cost, in some cases, is close to and even larger than the savings from the reduces communications, as empirical evidence has shown in [33]; (ii) Sparsification compressors and bias quantization compressor are not naively combatable with the efficient communication primitiveAll-Reduce due to their inherent structures, and they have to utilize All-Gather for aggregation in stead, which will significantly slow down the communication speed, as empirically shown in [1]; (iii) Some low-rank simplification compressors [29] and quantization compressors [28] [18] need to harness their uncompressed counterparts for warm-up at the early stage to stabilize the convergence, and the warm-up time is commonly nontrivial which to some extent renders their high compression ratios vacuous. Therefore, from a system-level perspective, the design ethos of a system-efficient communication-compression algorithm is that we should guarantee that the compression/decompression of the algorithm is computationally light and takes less time, and it should also be friendly to efficient collective communication primitives. Additionally, there is no need to resort to an uncompressed optimizer for warm-up.
To this end, we propose a 1-bit adaptive optimizer, called Binary randomization adaptive optimizer (Birder), which use the following updating rule is  $x_{t + 1} = x_t - \alpha_t\mathcal{Q}\left(\frac{m_t}{b_t}\right)$  where  $m_{t} = \beta m_{t - 1} + (1 - \beta)g_{t}$ ,  $b_{t} = \beta b_{t - 1} + (1 - \beta)|g_{t}|$  and  $g_{t}$  is the gradient, and  $\mathcal{Q}(\cdot)$  is a element-wise binary quantization operator. The main difference between Birder and existing gradient-quantization algorithms is that we directly quantize the entire adaptive update  $\frac{m_t}{b_t}$  rather than quantize the gradient  $g_{t}$  or the momentum  $m_{t}$ . Because  $-1\leq \frac{(m_t)_j}{(b_t)_j}\leq 1$ , where  $(m_t)_j$ ,  $(b_{t})_{j}$  are the  $j^{th}$  element of  $m_{t}$ ,  $b_{t}$  respectively, each element of  $\frac{m_t}{b_t}$  is easy to be randomly quantized to 1 or -1 in probability, making the quantization computationally light. Another advantage of Birder is that it does not require a full-precision optimizer to warm up at the early stage to ensure stable convergence. We also demonstrate Birder's convergence rate can match that of Adam. Moreover, taking into account the nature of Birder, we devise an efficient hierarchical communication scheme to further speed up communication, which sufficiently leverages the ultra-high intra-bandwidth among GPUs within the same node.
In particular, we make the following key contributions:
- We propose a novel 1-bit optimizer, dubbed Birder, which is a native communication-compression adaptive algorithm that element-wise quantizes the entire model update
and does not need to leverage its uncompressed counterpart for warm-up, making compression/decompression computationally light and the extreme quantization ratio exert its best function (Section 2).
- We theoretically prove that despite emolying extreme 1-bit quantization is employed, Birder still promise the same convergence speed as the full-precision Adam (Section 3).
- We develop a new communication scheme for 1-bit communication, called Hierarchical-1-bit-All-Reduce, which sufficiently harnesses the ultra-fast intra-connects to accelerate the local communication, and utilize more efficient commutation primitives to further reduce the communication overhead (Section 4).
- We perform extensive distributed training experiments to demonstrate the effectiveness of the proposed algorithm. As far as we know, running with DDP, our algorithm is the first work to consistently trump SGDM/Adam in terms of entire running time at little/no inference performance cost, reaching up to  $2.47 \times$  speedup for ResNet-50 and  $6.26 \times$  speedup for BERT-Base on 64 GPUs. (Section 5).
2 One-Bit Adaptive Optimizer Birder
In this section, we focus on solving the following problem for distributed training:
$$
\min  _ {x \in \mathbb {R} ^ {d}} f (x) = \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (x; \xi^ {(i)}) \tag {1}
$$
where  $x$  is the  $d$ -dimensional model parameter,  $n$  is the number of distributed workers.  $\xi^{(i)}$  is the sampled min-batch data on the  $i$ -th worker. The sampled min-batch data on all the workers is independent and identically distributed (i.i.d.).  $f_{i}(x; \xi^{(i)})$  is the loss function. Note that  $f_{i}(x; \xi_{i})$  is commonly abbreviated as  $f_{i}(x)$  in the following.
When distributedly training large-scale DNN models, using vanilla full-precision optimizers can cause communication bottleneck issues in gradient communication among workers at each iteration. To alleviate this problem, elegant SignSGD[4] was proposed, which merely takes the sign of each coordinate of the gradients. While this algorithm can substantially reduce the communication overhead, its practical performance is still inferior to popular adaptive optimizers, such as Adam. Fortunately, we observe that the mathematical formulations of SignSGD and Adam have close connections, providing an opportunity to propose a new optimizer that can combine their merits. This new optimizer can considerably reducing the communication volume with light computation, while maintaining fast convergence speed and high inference performance.
The mathematical updating rule of SignSGD can be formulated as:
$$
x _ {t + 1} \leftarrow x _ {t} - \alpha_ {t} \operatorname {S i g n} \left(g _ {t}\right) = x _ {t} - \alpha_ {t} \frac {g _ {t}}{\left| g _ {t} \right|} \tag {2}
$$
where  $\alpha_{t}$  is the learning rate,  $g_{t}$  denotes the estimated unbiased noisy gradient of  $f(x_{t})$  with random samples,  $\mathrm{Sign}(\cdot)$  is a element-wise signum, and  $|\cdot|$  is an element-wise absolute operator.
Whereas the updating rule of vanilla Adam [14] can be expressed as:
$$
m _ {t} \leftarrow \beta_ {1} m _ {t - 1} + (1 - \beta_ {1}) g _ {t},
$$
$$
v _ {t} \leftarrow \beta_ {2} v _ {t - 1} + (1 - \beta_ {2}) g _ {t} ^ {2}, \tag {3}
$$
$$
x _ {t + 1} \gets x _ {t} - \alpha_ {t} \frac {m _ {t}}{\sqrt {v _ {t}}},
$$
where  $\beta_{1}$  and  $\beta_{2}$  represents the exponential moving average factors  $^{3}$ .
If taking  $\beta_{1}$  and  $\beta_{2}$  to zero,  $\beta_{1},\beta_{2}\rightarrow 0$  in Eq. (3), Adam will be reduced to SignSGD.
For simplicity, we omit the bias correction for  $m_{t}$  and  $v_{t}$  and the small constant in the numerator.
Given the observations above, we propose a new optimizer that is an intermediate between SignSGD and Adam, referred to as Birder, i.e.,
$$
m _ {t} \leftarrow \beta m _ {t - 1} + (1 - \beta) g _ {t},
$$
$$
b _ {t} \leftarrow \beta b _ {t - 1} + (1 - \beta) | g _ {t} |, \tag {4}
$$
$$
x _ {t + 1} \leftarrow x _ {t} - \alpha_ {t} \mathcal {Q} \left(\frac {m _ {t}}{b _ {t}}\right),
$$
where the  $j$ -th elements of  $m_t, b_t$  rigorously satisfy  $-1 \leq \frac{(m_t)_j}{(b_t)_j} \leq 1$ ,  $\mathcal{Q}(\cdot)$  is an element-wise quantization operator, and it quantizes the  $j$ -th element of  $\frac{m_t}{b_t}$  as follows:
$$
\mathcal {Q} \left(\frac {\left(m _ {t}\right) _ {j}}{\left(b _ {t}\right) _ {j}}\right) = \left\{ \begin{array}{l l} 1, & \text {w i t h p r o b a b i l i t y} p = \frac {1}{2} \left(\frac {\left(m _ {t}\right) _ {j}}{\left(b _ {t}\right) _ {j}} + 1\right) \\ - 1, & \text {w i t h p r o b a b i l i t y} 1 - p \end{array} , \right. \tag {5}
$$
where  $\mathbb{E}\left(\mathcal{Q}\left(\frac{m_t}{b_t}\right)\right) = \frac{m_t}{b_t}$ , so  $\mathcal{Q}(\cdot)$  is unbiased, and proof is provided in Section A of the appendix. The detailed implementation of Birder in a parameter-server model is illustrated in Algorithm 1.
Algorithm 1. Birder
1: Input: all workers's model parameter  $x_0, x_1$ , the  $i^{th}$  worker's momentum  $m_0^{(i)} = 0$ ,  $b_0^{(i)} = 0$ , the  $i^{th}$  worker's local error  $e_0^{(i)} = 0$ , server's global error  $\bar{e}_0 = 0$ , exponential moving average factor  $\beta$ , the threshold  $T_0$ , and the learning rate sequence  $\{\alpha_t\}$ .
2: for  $t = 1, \dots, T$  do
3: (On the  $i^{th}$  worker)
4: Randomly sample  $\xi_{t}^{(i)}$  and compute local gradient:  $g_{t}^{(i)} = \nabla f_{i}(x_{t};\xi_{t}^{(i)})$
5: Update the local  $m_t^{(i)}$ :  $m_t^{(i)} = \beta m_{t - 1}^{(i)} + (1 - \beta)g_t^{(i)}$
6: Update the local  $\hat{b}_t^{(i)}\colon \hat{b}_t^{(i)} = \beta \hat{b}_{t - 1}^{(i)} + (1 - \beta)|g_t^{(i)}|$
7: Update the local  $b_{t}^{(i)}$ : if  $t > T_0$  {  $b_{t}^{(i)} = \max(b_{t-1}^{(i)}, \hat{b}_{t}^{(i)})$  } else {  $b_{t}^{(i)} = \hat{b}_{t}^{(i)}$  }
8: Quantize the local update:  $u_{t}^{(i)} = \mathcal{Q}\left(\frac{m_{t}^{(i)}}{b_{t}^{(i)}} + e_{t - 1}^{(i)}\right)$
9: Update the local error feedback  $e_t^{(i)} : e_t^{(i)} = e_{t-1}^{(i)} + \frac{m_t^{(i)}}{b_t^{(i)}} - u_t^{(i)}$
10: Send  $u_{t}^{(i)}$  to the server
11: (On server)
12: Average all received  $q_{t}$  and quantize it:  $\bar{u}_{t} = \mathcal{Q}\left(\frac{1}{n}\sum_{i = 1}^{n}u_{t}^{(i)} + \bar{e}_{t - 1}\right)$
13: Update the global error feedback  $\bar{e}_t:\bar{e}_t = \bar{e}_{t - 1} + \frac{1}{n}\sum_{i = 1}^{n}u_t^{(i)} - \bar{u}_t$
14: Send back  $\bar{u}_t$  to all workers
15: (On the  $i^{th}$  worker)
16: Update the local model parameter  $x_{t + 1}$ :  $x_{t + 1} = x_t - \alpha_t\bar{u}_t$
17: end for
The appealing characters of Birder are summarized in the following:
- Compared to SGD-type optimizers, Adam provides a fast convergence rate in practice by adaptively preconditioning the gradients with  $v_{t}$ . Birder inherits this feature to accelerate convergence speed. On the other hand, unlike Adam, Birder employs the same exponential moving average factor  $\beta$  for both  $m_{t}$  and  $b_{t}$ . This eliminates the need for bias correction and reduces the amount of tuning work required.
- Existing quantization optimizers are built upon vector-wise quantization [3][19][22] Besides the quantization, they also require heavy computations to compute the norm of a vector
$$
\mathcal {Q} \left((g _ {t}) _ {j}\right) = \left\{ \begin{array}{l l} \| g _ {t} \| _ {p} \mathrm {s i g n} ((g _ {t}) _ {j}) \cdot \frac {r}{s}, & \text {w i t h p r o b a b i l i t y} p _ {i} = \frac {s | (g _ {t}) _ {j} |}{\| g _ {t} \| _ {p}} - r \\ \| g _ {t} \| _ {p} \mathrm {s i g n} ((g _ {t}) _ {j}) \cdot \frac {r + 1}{s}, & \text {w i t h p r o b a b i l i t y} 1 - p _ {i} \end{array} \right.
$$
where  $\| g_t\| _p = \left(\sum_j|(g_t)_j|^p\right)^{\frac{1}{p}}(p\geq 1),0\leq r < s(r,l\in \mathbb{N})$  and  $\frac{s|(g_t)_j|}{\|g_t\|_2}\in [\frac{r}{s},\frac{r + 1}{s} ]$
This step follows the technique in AMSGrad [20]. It is more about theoretical significance, and we commonly do not implement it in practice.
4The typical gradient-quantized optimizer QSGD quantizes the gradient as follows:
and estimate the sign of each element, which will render the saved communication time cost somewhat meaningless. In contrast, Birder element-wise quantizes the update, and the subtle design for  $m_t$  and  $b_t$  ensures the unquantized update is strictly bounded in the range  $[-1, 1]$ , allowing quantization is computed easily and lightly. Further, unlike most quantization optimizers that only compress the gradients [6], Birder performs the quantization for the entire adaptive update, which further streamlines the optimization process.
Remark. We have noticed that the prior works 1-bit Adam ([28]) and its variants ([16], [18]) are also categorized as 1-bit adaptive optimizers. However, the design ethos of 1-bit Adam and Birder differ significantly. 1-bit Adam is still built on gradient quantization and and essentially functions as a preconditioned SGDM. 1-bit Adam runs full-precision Adam in the beginning (warm-up phase) and utilizes it as a fixed precondition for SGDM during the rest of training (compression phase). There are three aspects that influence 1-bit Adam to indeed accelerate communication. First, the warm-up stage constitutes approximately  $15\% - 25\%$  of the total steps, which to some extent discounts the high quantization ratio. Second, the vector-wise quantization employed by 1-bit Adam necessitates extra computations, including the calculation of vector norms and estimation of element signs, which are then transmitted as additional data. These factors diminish the time savings achieved through reduced communication bits. Third, the vector-wise quantization technique employed by 1-bit Adam is not compatible with the common-used distributed framework DDP (system-level engineered distributed framework). In DDP, communication data is uniformly divided into buckets of equal size on the sender's side to enhance communication efficiency. Consequently, when vector-wise quantization is used, communication data from a single layer may be divided into different buckets, resulting in substantial errors during restoration on the receiver's end.
3 Theoretical Analysis
In this section, we present the theoretical convergence guarantee for Birder (Algorithm 1). We first introduce some necessary assumptions.
Assumption 1.[Bounded infimum] For any  $x$  and a constant  $f^*$ , we have the objective value  $f(x) \geq f^*$ .
Assumption 2. [Lipschitz continuous gradient] The gradient  $\nabla f(\cdot)$  is  $L$ -Lipschitz continuous, i.e.,  $\| \nabla f(x) - \nabla f(y)\| \leq L\| x - y\| _2$ ,  $\forall x,y\in \mathbb{R}^d$
Assumption 3. [Unbias and independent noisy gradient] The gradient with respect to the random samples on each worker and at a different time is independent identically distributed (i.i.d.), i.e.,  $\mathbb{E}[g_t^{(i)}] = \nabla f(x_t), \forall t \geq 1$ ,  $g_t^{(i)}$  is independent of  $g_t^{(j)}$  for  $i \neq j$ , and  $g_{t_1}^{(i)}$  is independent of  $g_{t_2}^{(j)}$  for  $t_1 \neq t_2$ .
Assumption 4. [Bounded gradient] The noisy gradient and the full-set gradient are bounded i.e.,  $\| g_t^{(i)}\| \leq G,\| \nabla f_t(x)\| \leq G,\forall t\geq 1.$
Under the assumptions above, we then present the theoretical convergence for Birder in Algorithm 1.
Theorem 1. For Birder in Algorithm 1, under Assumption 1-4, assuming  $(b_{t}^{(i)})_{j}\geq \rho >0$ $\forall j\in [1,2,\dots,d]^5$  , choosing  $\alpha_{t} = \frac{c}{\sqrt{t}}$ $\forall t\in [1,2,\dots,T]$  and  $\alpha_0 = \alpha_1$  , and defining  $z_{1} = x_{1} + \alpha_{1}(\delta_{1} - e_{1})$  where  $\delta_1 = \frac{1}{n}\sum_{i = 1}^n{\frac{m_1^{(i)}}{b_1^{(i)}}} - \frac{\sum_{i = 1}^n m_1^{(i)}}{\sum_{i = 1}^n b_1^{(i)}}$  and  $e_1 = \frac{1}{n}\sum_{i = 1}^{n}e_1^{(i)} + \bar{e}_1$  we then have the following
$$
\mathbb {E} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} \| \nabla f (x _ {t}) \| \right] ^ {2} \leq \frac {C _ {1}}{\sqrt {T}} + \frac {C _ {2} (1 + \log T)}{\sqrt {T}}, \tag {6}
$$
$$
C _ {1} = c G \left(\mathbb {E} [ f (z _ {1}) - f ^ {*} ] + \frac {3 c ^ {2} d L}{1 6} + \frac {\beta c d G ^ {2}}{(1 - \beta) \rho} + \frac {4 c d G ^ {2}}{\rho} + \frac {c ^ {2} \beta^ {2} L G ^ {2} d}{\rho^ {2} (1 - \beta) ^ {2}}\right),
$$
$$
C _ {2} = c ^ {3} G \left(\frac {(8 \beta^ {2} + 1 0 \beta + 5) L ^ {2} d}{(1 - \beta) ^ {2}} + \frac {G ^ {2} (1 + L)}{2 \rho^ {2}} + 2 d L\right).
$$
<sup>5</sup>We commonly add a small constant to  $b_{t}$  to avoid zero denominators for numerical stability, which guarantees this assumption holds in practice.
Figure 1: Paradigm of Hierarchical-1-bit-All-Reduce
The theoretical results suggested that 1-bit Birder essentially achieve the same convergence rate  $(O\left(\frac{1}{\sqrt{T}}\right))$  as the uncompressed Adam [14][8].
4 Hierarchical-1-bit-All-Reduce
The data communication for Birder is one-bit, which cannot be directly aggregated using the efficient All-Reduce. Additionally, there is a significant imbalance between the intra-node and inter-node bandwidths. If we attempt to aggregate the data uniformly from both intra-nodes and inter-nodes, the communication process will be hindered by the inter-node data exchanges, resulting in slower overall communication.
In light of the problems above, we propose a hierarchical communication scheme called Hierarchical-1-bit-All-Reduce. This scheme efficiently aggregates our 1-bit data by leveraging the ultra-high intra-node bandwidth and reducing the inter-node communication overhead. Assuming we have  $n$  nodes, each containing  $m$  GPUs, and the overall volume for each GPU needs to be communicated is  $P$ , as visually depicted in Figure 1, the steps of Hierarchical-1-bit-All-Reduce are as follows: (i) Each GPU performs Reduce-Scatter to locally aggregate and scatter the data within its node. The communication volume for each GPU in this step is  $\frac{(m - 1)P}{m}$ . (ii) Each GPU then applies Birder to quantize the data, resulting in a reduced volume of  $\frac{P}{32m}$  on each GPU. (iii) The GPU proceeds with 1-bit-All-Reduce to inter-aggregate the data. This step consists of two sub-steps: 1) Each GPU performs All-to-All to collect the corresponding data from GPUs in other nodes, with a communication volume of  $\frac{(n - 1)P}{32mn}$ . 2) Each GPU averages and re-quantizes the data, followed by All-Gather operation to gather the data. The communication volume in this sub-step is also  $\frac{(n - 1)P}{32mn}$ . (iv) Finally, each GPU performs All-Gather to intra-aggregate the data, with a communication volume of  $\frac{(m - 1)P}{32m}$ .
Compared to the time cost of inter-node communication, the time cost of intra-node communication is relatively insignificant. Thus, when utilizing Hierarchical-1-bit-All-Reduce, the majority of the communication cost arises from the 1-bit All-Reduce step in Step (iii). The communication volume across nodes for all GPUs in this scheme is approximately  $\frac{2(n - 1)P}{32}$ . In contrast, if we were to simply employ the original All-Gather to aggregate data, the communication volume across nodes for all GPUs would be approximately  $\frac{m^2n(n - 1)P}{32}$ . Consequently, Hierarchical-1-bit-All-Reduce proves significantly more efficient than the original All-Gather.
Notably, the work [32] also introduces a 1-bit data communication scheme among nodes, but it can only guarantee the expected value of the gathered 1-bit data equals to the average of the original 1-bit data among nodes, thereby it will bring performance deterioration. In contrast, All-to-All in Hierarchical-1-bit-All-Reduce ensures the final data exactly equals to the average of the original data.
(a) Epoch-wise, ResNet-50, batch size  $= 32 \times 64$
(b) Time-wise, ResNet-50, batch size  $= 32 \times 64$
(c) Epoch-wise, ResNet-50, batch size  $= 128 \times 64$
(d) Time-wise, ResNet-50, batch size  $= 128 \times 64$
(e) Epoch-wise, Bert-Base, batch size  $= 3 \times 64$