Title: Birder: Communication-Efficient 1-bit Adaptive Optimizer for Practical Distributed DNN Training
Venue: NeurIPS 2023
Accepted: yes
Keywords: optimizer

1-bit optimizer

distributed learning

optimization for deep networks

communication efficiency
Abstract: Various gradient compression algorithms have been proposed to alleviate the communication bottleneck in distributed learning, and they have demonstrated effectiveness in terms of  high compression ratios and theoretical low communication complexity. However, when it comes to practically training modern deep neural networks (DNNs),  these algorithms have yet to match the inference performance of uncompressed SGD-momentum  (SGDM) and adaptive optimizers (e.g.,Adam). More importantly,  recent studies suggest that these algorithms actually offer no speed advantages over SGDM/Adam when used with common  distributed DNN training frameworks ( e.g., DistributedDataParallel (DDP)) in the typical settings, due to heavy compression/decompression computation or incompatibility with the efficient All-Reduce or the requirement of uncompressed warmup at the early stage. For these reasons, we propose a novel 1-bit adaptive optimizer, dubbed *Bi*nary *r*andomization a*d*aptive optimiz*er* (**Birder**). The quantization of Birder can be easily and lightly computed, and it does not require warmup with its uncompressed version in the beginning. Also, we devise Hierarchical-1-bit-All-Reduce to further lower the communication volume. We theoretically prove that it promises the same convergence rate as the Adam. Extensive experiments, conducted on 8 to 64 GPUs (1 to 8 nodes) using DDP, demonstrate that Birder achieves comparable inference performance to uncompressed SGDM/Adam, with up to ${2.5 \times}$ speedup for training ResNet-50 and ${6.3\times}$ speedup for training BERT-Base. Code is publicly available at https://openi.pcl.ac.cn/c2net_optim/Birder.

Birder : Communication-Efficient 1-bit Adaptive Optimizer for Practical Distributed DNN Training

Abstract

Various gradient compression algorithms have been proposed to alleviate the communication bottleneck in distributed learning, and they have demonstrated effectiveness in terms of high compression ratios and theoretical low communication complexity. However, when it comes to practically training modern deep neural networks (DNNs), these algorithms have yet to match the inference performance of uncompressed SGD-momentum ( SGDM ) and adaptive optimizers ( e.g. , Adam ). More importantly, recent studies suggest that these algorithms actually offer no speed advantages over SGDM/Adam when used with common distributed DNN training frameworks ( e.g. , DistributedDataParallel (DDP) ) in the typical settings, due to heavy compression/decompression computation or incompatibility with the efficient All-Reduce or the requirement of uncompressed warmup at the early stage. For these reasons, we propose a novel 1-bit adaptive optimizer, dubbed Bi nary r andomization a d aptive optimiz er ( Birder ). The quantization of Birder can be easily and lightly computed, and it does not require warmup with its uncompressed version in the beginning. Also, we devise Hierarchical-1-bit-All-Reduce to further lower the communication volume. We theoretically prove that it promises the same convergence rate as the Adam . Extensive experiments, conducted on 8 to 64 GPUs (1 to 8 nodes) using DDP , demonstrate that Birder achieves comparable inference performance to uncompressed SGDM/Adam , with up to 2 . 5 × speedup for training ResNet-50 and 6 . 3 × speedup for training BERT-Base. Code is publicly available at https://openi.pcl.ac.cn/c2net_optim/Birder.

1 Introduction

With the rapid development of computational power, "bigger" and "bigger" deep neural network (DNN) models are proposed for expect better performance, from the early classical models, such as AlexNet(61 million parameters) [15], and ResNet (ResNet-50: 20.5 million parameters) [12] to the current large language models (LLMs), such as BERT (BERT-Lagre: 340 million parameters )[10], and GPT (GPT-3: 176 billion parameters)[5]. Scalable parallelism across distributed computing workers for training these large-scale models becomes a necessity. During training, millions to billions of parameters need to be communicated among workers at each iteration, so distributed large-scale DNN training almost invariably suffers from the communication bottleneck.

To address the communication bottleneck, a wide variety of lossy gradient compression algorithms have been proposed to lower the communication overhead. The algorithms can be broadly divided

into three categories based on the compression techniques, including low-precision approximation ( e.g. , SignSGD [4], TernGrad [31], and QSGD [3], 1-bit Adam [28]), low-rank simplification ( e.g. , ATOMO [30], PowerSGD [29], and GradZip [9]), and sparsification ( e.g. , Randomk [27], Topk [2], and MSTopk [24]). In addition to the specific compression techniques, some other works, such as Error Feedback(EF) [23][26], EF21 [21], DIANA [19] and MARINA [11], focus on changing the compression objects from the gradient to the gradient and delayed error summation, or the gradient differences to mitigate compressing errors and/or accelerate the convergence rate.

Gradient compression algorithms have demonstrated promising results with a high compression ratio and low oracle/communication complexity in theory. However, when practically training DNNs, they are still inferior to uncompressed SGDM/Adam in terms of inference performance. This is because, these gradient compression algorithms are SGD-type optimizers, which will be commonly reduced to vanilla SGD without momentum if compression is not employed. The performance for a compressed optimizer is commonly upper bounded by its uncompressed counterpart, while vanilla SGD is typically less effective than SGDM for training DNNs. Particularly , SGD-type optimizers are known to be substantially inferior to adaptive optimizers ( e.g. , Adam ) for training Transformer-based networks [17] [34] [7], which have become predominant in the DNN community. Supporting empirical evidences for this phenomenon can be found in Section B in the Appendix. Furthermore, if we apply the techniques of gradient compression algorithms to compress and communicate the gradients, and subsequently utilize the compressed gradients to construct adaptive optimizers in the local nodes, the final performance will be degraded [28]. Therefore, designing native communication-compression adaptive optimizers is an underexplored problem that requires further research.

As for the system-level speed, recent studies ([33],[1]) pointed out, when distributedly training typical DNN models ( e.g. , ResNet-50 and BERT-Base) with off-the-shelf DistributedDataParallel (DDP) at typical bandwidths ( e.g. , 10Gbps), existing gradient compression algorithms are still slower than uncompressed SGDM/Adam . This is because, the compressor for these algorithms are either quantization or sparsification or low-rank simplification, which exhibit one or more weaknesses below. ( i ) Vector-wise quantization and low-rank simplification compressors are commonly computationally heavy, and their time cost, in some cases, is close to and even larger than the savings from the reduces communications, as empirical evidence has shown in [33] ; ( ii ) Sparsification compressors and bias quantization compressor are not naively combatable with the efficient communication primitive AllReduce due to their inherent structures, and they have to utilize All-Gather for aggregation in stead, which will significantly slow down the communication speed, as empirically shown in [1] ; ( iii ) Some low-rank simplification compressors [29] and quantization compressors [28] [18] need to harness their uncompressed counterparts for warm-up at the early stage to stabilize the convergence, and the warmup time is commonly nontrivial which to some extent renders their high compression ratios vacuous. Therefore, from a system-level perspective, the design ethos of a system-efficient communicationcompression algorithm is that we should guarantee that the compression/decompression of the algorithm is computationally light and takes less time, and it should also be friendly to efficient collective communication primitives. Additionally, there is no need to resort to an uncompressed optimizer for warm-up.

To this end, we propose a 1-bit adaptive optimizer, called Bi nary r andomization a d aptive optimiz er ( Birder ), which use the following updating rule is x t +1 = x t -α t Q ( m t b t ) where m t = βm t -1 +(1 -β ) g t , b t = βb t -1 +(1 -β ) | g t | and g t is the gradient, and Q ( · ) is a element-wise binary quantization operator. The main difference between Birder and existing gradient-quantization algorithms is that we directly quantize the entire adaptive update m t b t rather than quantize the gradient g t or the momentum m t . Because -1 ≤ ( m t ) j ( b t ) j ≤ 1 , where ( m t ) j , ( b t ) j are the j th element of m t , b t respectively, each element of m t b t is easy to be randomly quantized to 1 or -1 in probability, making the quantization computationally light. Another advantage of Birder is that it does not require a full-precision optimizer to warm up at the early stage to ensure stable convergence. We also demonstrate Birder 's convergence rate can match that of Adam . Moreover, taking into accost the nature of Birder , we devise an efficient hierarchical communication scheme to further speed up communication, which sufficiently leverages the ultra-high intra-bandwidth among GPUs within the same node.

In particular, we make the following key contributions :

We propose a novel 1-bit optimizer, dubbed Birder , which is a native communicationcompression adaptive algorithm that element-wise quantizes the entire model update

and does not need to leverage its uncompressed counterpart for warm-up , making compression/decompression computationally light and the extreme quantization ratio exert its best function (Section 2).

We theoretically prove that despite emolying extreme 1-bit quantization is employed, Birder still promise the same convergence speed as the full-precision Adam (Section 3).

We develop a new communication scheme for 1-bit communication, called Hierarchical-1bit-All-Reduce , which sufficiently harnesses the ultra-fast intra-connects to accelerate the local communication, and utilize more efficient commutation primitives to further reduce the communication overhead (Section 4).

We perform extensive distributed training experiments to demonstrate the effectiveness of the proposed algorithm. As far as we know, running with DDP , our algorithm is the first work to consistently trump SGDM/Adam in terms of entire running time at little/no inference performance cost , reaching up to 2 . 47 × speedup for ResNet-50 and 6 . 26 × speedup for BERT-Base on 64 GPUs. (Section 5).

2 One-Bit Adaptive Optimizer Birder

In this section, we focus on solving the following problem for distributed training :

where x is the d -dimensional model parameter, n is the number of distributed workers. ξ ( i ) is the sampled min-batch data on the i -the worker. The sampled min-batch data on all the workers is independent and identically distributed ( i.i.d. ). f i ( x ; ξ ( i ) ) is the loss function. Note that f i ( x ; ξ i ) is commonly abbreviated as f i ( x ) in the following.

When distributedly training large-scale DNN models, using vanilla full-precision optimizers can cause communication bottleneck issues in gradient communication among workers at each iteration. To alleviate this problem, elegant SignSGD [4] was proposed, which merely takes the sign of each coordinate of the gradients. While this algorithm can substantially reduce the communication overhead, its practical performance is still inferior to popular adaptive optimizers, such as Adam . Fortunately, we observe that the mathematical formulations of SignSGD and Adam have close connections, providing an opportunity to propose a new optimizer that can combine their merits. This new optimizer can considerably reducing the communication volume with light computation, while maintaining fast convergence speed and high inference performance.

The mathematical updating rule of SignSGD can be formulated as:

where α t is the learning rate, g t denotes the estimated unbias noisy gradient of f ( x t ) with random samples, Sign( · ) is a element-wise signum, and | · | is an element-wise absolute operator.

Whereas the updating rule of vanilla Adam [14] can be expressed as:

where β 1 and β 2 represents the exponential moving average factors 3 .

If taking β 1 and β 2 to zero, β 1 , β 2 → 0 in Eq. (3), Adam will be reduced to SignSGD .

Given the observations above, we propose a new optimizer that is an intermediate between SignSGD and Adam , referred to as Birder , i.e. ,

where the j -th elements of m t , b t rigorously satisfies -1 ≤ ( m t ) j ( b t ) j ≤ 1 , Q ( · ) is an element-wise quantization operator, and it quantizes the j -th element of m t b t as follows:

where E ( Q ( m t b t )) = m t b t , so Q ( · ) is unbiased, and proof is provided in Section A of the appendix. The detailed implementation of Birder in a parameter-server model is illustrated in Algorithm 1.

Algorithm 1. Birder

1: Input : all workers's model parameter x 0 , x 1 , the i th worker's momentum m ( i ) 0 = 0 , b ( i ) 0 = 0 , the i th worker's local error e ( i ) 0 = 0 , server's global error ¯ e 0 = 0 , exponential moving average factor β , the threshold T 0 , and the learning rate sequence { α t } .

2: for t = 1 , ..., T do

3: ( On the i th worker )

4: Randomly sample ξ ( i ) t and compute local gradient: g ( i ) t = ∇ f i ( x t ; ξ ( i ) t )

5: Update the local m ( i ) t : m ( i ) t = βm ( i ) t -1 +(1 -β ) g ( i t

)

6: Update the local ˆ b ( i ) t : ˆ b ( i ) t = β ˆ b ( i ) t -1 +(1 -β ) | g ( i ) t |

7: Update the local b ( i ) t : if t > T 0 { b ( i ) t = max( b ( i ) t -1 , ˆ b ( i ) t ) } else { b ( i ) t = ˆ b ( i ) t } *

8: Quantize the local update: u ( i ) t = Q ( m ( i ) t b ( i ) + e ( i ) t -1 )

10: Send u ( i ) t to the server

t 9: Update the local error feedback e ( i ) t : e ( i ) t = e ( i ) t -1 + m ( i ) t b ( i ) t -u ( i ) t

11: ( On server )

12: Average all received q t and quantize it: ¯ u t = Q ( 1 n ∑ n i =1 u ( i ) t + ¯ e t -1 )

14: Send back ¯ u t to all workers

13: Update the global error feedback ¯ e t : ¯ e t = ¯ e t -1 + 1 n ∑ n i =1 u ( i ) t -¯ u t

15: ( On the i th worker )

16: Update the local model parameter x t +1 : x t +1 = x t -α t ¯ u t

17: end for

*

The appealing characters of Birder are summarized in the following:

Compared to SGD-type optimizers, Adam provides a fast convergence rate in practice by adaptively preconditioning the gradients with v t . Birder inherits this feature to accelerate convergence speed. On the other hand, unlike Adam , Birder employs the same exponential moving average factor β for both m t and b t . This eliminates the need for bias correction and reduces the amount of tuning work required.

Existing quantization optimizers are built upon vector-wise quantization [3][19][22] 4 Besides the quantization, they also require heavy computations to compute the norm of a vector

glyph[negationslash]

and estimate the sign of each element, which will renders the saved communication time cost somewhat meaningless. In contrast, Birder element-wise quantizes the update, and the subtle design for m t and b t ensures the unquantized update is strictly bounded in the range [ -1 , 1] , allowing quantization is computed easily and lightly . Further, unlike most quantization optimizers that only compress the gradients [6], Birder performs the quantization for the entire adaptive update , which further streamlines the optimization process.

Remark. We have noticed that the prior works 1-bit Adam ([28]) and its variants ([16], [18]) are also categorized as 1-bit adaptive optimizers. However, the design ethos of 1-bit Adam and Birder differ significantly. 1-bit Adam is still built on gradient quantization and and essentially functions as a preconditioned SGDM . 1-bit Adam runs full-precision Adam in the beginning (warm-up phase) and utilizes it as a fixed precondition for SGDM during the rest of training (compression phase). There are three aspects that influence 1-bit Adam to indeed accelerate communication. First , the warm-up stage constitutes approximately 15%-25% of the total steps, which to some extent discounts the high quantization ratio. Second , the vector-wise quantization employed by 1-bit Adam necessitates extra computations, including the calculation of vector norms and estimation of element signs, which are then transmitted as additional data. These factors diminish the time savings achieved through reduced communication bits. Third , the vector-wise quantization technique employed by 1-bit Adam is not compatible with the common-used distributed framework DDP (system-level engineered distributed framework). In DDP , communication data is uniformly divided into buckets of equal size on the sender's side to enhance communication efficiency. Consequently, when vector-wise quantization is used, communication data from a single layer may be divided into different buckets, resulting in substantial errors during restoration on the receiver's end.

3 Theoretical Analysis

In this section, we present the theoretical convergence guarantee for Birder (Algorithm 1). We first introduce some necessary assumptions.

Assumption 1. [Bounded infimum] For any x and a constant f ∗ , we have the objective value f ( x ) ≥ f ∗ .

Assumption 2. [Lipschitz continuous gradient] The gradient ∇ f ( · ) is L -Lipschitz continuous, i.e., , ‖∇ f ( x ) -∇ f ( y ) ‖ ≤ L ‖ x -y ‖ 2 , ∀ x, y ∈ R d .

glyph[negationslash]

Assumption 3. [Unbias and indpendent noisy gradient] The gradient with respect to the random samples on each worker and at a different time is independent identically distributed (i.i.d.), i.e., E [ g ( i ) t ] = ∇ f ( x t ) , ∀ t ≥ 1 , g ( i ) t is independent of g ( j ) t for i = j , and g ( i ) t 1 is independent of g ( j ) t 2 for t 1 = t 2 .

Assumption 4. [Bounded gradient] The noisy gradient and the full-set gradient are bounded i.e., ‖ g ( i ) t ‖ ≤ G, ‖∇ f t ( x ) ‖ ≤ G, ∀ t ≥ 1 .

Under the assumptions above, we then present the theoretical convergence for Birder in Algorithm 1.

Theorem 1. For Birder in Algorithm 1, under Assumption 1-4, assuming ( b ( i ) t ) j ≥ ρ > 0 , ∀ j ∈ [1 , 2 , ..., d ] 5 , choosing α t = c √ t , ∀ t ∈ [1 , 2 , ..., T ] and α 0 = α 1 , and defining z 1 = x 1 + α 1 ( δ 1 -e 1 )

where δ 1 = 1 n ∑ n i =1 m ( i ) 1 b ( i ) 1 -∑ n i =1 m ( i ) 1 ∑ n i =1 b ( i ) 1 and e 1 = 1 n ∑ n i =1 e ( i ) 1 + ¯ e 1 , we then have the following

where

Intra-nodecommunication

Nocommunication

Inter-nodecommunication

32-bit

32-bit

1-bit

1-bit

1-bit

GPU1

Reduce-

GPU2

Quantize

All-Gather

GPU4

GPUS

Reduce-

Quantize

1-bit All-

All-Gather

Reduce

GPU8

GPU9

Reduce

GPU10

Quantize

All-Gather

GPU11

GPU12

1-bit All-

Reduce

Average&

GPU1

GPU1

Requantize

GPU1

GPU1

1日日

GPU5

All-to-All

GPU5

Average&

Requantize

GPU5

ALL-Gather

GPUS

日日

Average&

GPU 9

GPU9

Requantize

GPU9

GPU9

1日日

The theoretical results suggested that 1-bit Birder essentially achieve the same convergence rate ( O ( 1 √ T ) ) as the uncompressed Adam [14][8].

4 Hierarchical-1-bit-All-Reduce

The data communication for Birder is one-bit, which cannot be directly aggregated using the efficient All-Reduce . Additionally, there is a significant imbalance between the intra-node and inter-node bandwidths. If we attempt to aggregate the data uniformly from both intra-nodes and inter-nodes, the communication process will be hindered by the inter-node data exchanges, resulting in slower overall communication.

In light of the problems above, we propose a hierarchical communication scheme called Hierarchical1-bit-All-Reduce . This scheme efficiently aggregates our 1-bit data by leveraging the ultra-high intra-node bandwidth and reducing the inter-node communication overhead. Assuming we have n nodes, each containing m GPUs, and the overall volume for each GPU needs to be communicated is P ,as visually depicted in Figure 1, the steps of Hierarchical-1-bit-All-Reduce are as follows: ( i ) Each GPU performs Reduce-Scatter to locally aggregate and scatter the data within its node. The communication volume for each GPU in this step is ( m -1) P m . ( ii )Each GPU then applies Birder to quantize the data, resulting in a reduced volume of P 32 m on each GPU. ( iii ) The GPU proceeds with 1bit-All-Reduce to inter-aggregate the data. This step consists of two sub-steps: 1) Each GPU performs All-to-All to collect the corresponding data from GPUs in other nodes, with a communication volume of ( n -1) P 32 mn . 2) Each GPU averages and re-quantizes the data, followed by All-Gather operation to gather the data. The communication volume in this sub-step is also ( n -1) P 32 mn . (iv)Finally, each GPU performs All-Gather to intra-aggregate the data, with a communication volume of ( m -1) P 32 m .

Compared to the time cost of inter-node communication, the time cost of intra-node communication is relatively insignificant. Thus, when utilizing Hierarchical-1-bit-All-Reduce , the majority of the communication cost arises from the 1-bit All-Reduce step in Step ( iii ). The communication volume across nodes for all GPUs in this scheme is approximately 2( n -1) P 32 . In contrast, if we were to simply employ the original All-Gather to aggregate data, the communication volume across nodes for all GPUs would be approximately m 2 n ( n -1) P 32 . Consequently, Hierarchical-1-bit-All-Reduce proves significantly more efficient than the original All-Gather .

Notably, the work [32] also introduces a 1-bit data communication scheme among nodes, but it can only guarantee the expected value of the gathered 1-bit data equals to the average of the original 1-bit data among nodes, thereby it will bring performance deterioration. In contrast, All-to-All in Hierarchical-1-bit-All-Reduce ensures the final data exactly equals to the average of the original data.

SGDM

SGD

1-bit Adam

1-bit Adam

Birder

Birder

3

5S07

5S07

Q

20

40

60

80

100

Q

0

100

200

300

400

Epoch

Wall clock time (min)

SGDM

SGDM

1-bit Adam

1-bit Adam

Birder

Birder

3

SS07

SS07

20

40

60

80

100

L

50

100

Epoch

Wall clock time (min)

BertAdam

BertAdam

2.5

1-bit Adam

2.5

1-bit Adam

Birder

Birder

2.0 -

2.0 -

SSO7

1.5

SSO7

1.5

1.0 -

1.0

0.5

0.5

0.0

0.0

0.5

1.0

1.5

2.0

0.0 -

0

50

100

150

Epoch

Wall clock time (min)

(e) Epoch-wise, Bert-Base, batch size= 3 × 64

(f) Time-wise, Bert-Base, batch size= 3 × 64

5 Experiments

Recently, several works [33],[1] have shown that when utilizing the system-level engineered distributed data-parallel framework DDP , the existing communication-compression optimizers (excluding 1-bit Adam ) still perform slower than the uncompressed SGDM/Adam . Therefore, in our evaluation, we focus on assessing the performance of Birder , the uncompressed SGDM/Adam , and the closely related algorithm 1-bit Adam through distributed training experiments using the benchmark models ResNet-50 (CNN) and BERT-Base (Transformer). More extensive experiments can be founded in Section B of the appendix.

5.1 Experimental Settings

Our experiments were conducted on a testbed consisting of 1, 2, 4, 8 nodes interconnected via 10Gbps Ethernet. Each node was equipped with 8 Nvidia Tesla A100-80GB GPUs. The hardware and software configurations were identical across all instances, with Ubuntu 20.04.4 LTS serving as the operating system. PyTorch 1.11.0 was used as the primary framework, accompanied by CUDA-11.6, cuDNN-8.2, NCCL-2.10.3, and PyTorch 1.11.0 for other relevant libraries. Notably, to ensure compatibility with PyTorch's DDP , certain components of Birder and our hierarchical communication scheme were implemented within the customized communication hook of DDP .

Training details. For the experiments over ResNet-50, we evaluate the convergence and performance of SGDM , 1-bit Adam and Birder on ILSVRC2012. The batch size per GPU is set to 32 or 128 with the standard input resolution 224 × 224 . When employing SGDM (baseline) , the learning rate starts at 0 . 1 × batch size 256 with momentum of 0 . 9 and weight decay of 0 . 0001 . When employing 1-bit Adam and Birder , the learning rate starts at 0 . 001 × batch size 256 with weight decay of 0 . 0001 , and [ β 1 , β 2 ] for 1-bit Adam is set to [0 . 9 , 0 . 999] and β for Birder is set to 0 . 95 . Then, the learning rate is divided by 10 after 30 , 60 and 90 epochs, and training is finally terminated after 100 epochs. Specifically, the first 15 epochs are used as the warmup stage for 1-bit Adam . For the experiments over BERT-Base, we access the convergence and performance of BertAdam (baseline), 1-bit Adam and Birder for SQuAD 1.1 fine-tuning task using a pre-trained BERT-Base model checkpoint from HuggingFace 6 . The batch size per GPU is set to 3 . We perform fine-tuning for 2 epochs. The learning rate linearly increases to 1 × 10 -4 steps in the early 500 steps and then linearly decreases to 0 in the rest iteration. Specifically, the first 0 . 2 × steps are used as the warmup stage for 1-bit Adam . [ β 1 , β 2 ] for BertAdam , and 1-bit Adam is set to [0 . 9 , 0 . 999] and β for Birder is set to 0 . 9 .

5.2 Experimental Results

Figure 2 shows the convergence behaviors of epoch-wise and time-wise training for SGDM / BertAdam (baseline), 1-bit Adam , and Birder using ResNet-50 and BERT-Base models running on 64 GPUs. The experimental results clearly demonstrate that Birder achieves a similar epoch-wise convergence rate compared to the baseline. However, the actual training speed of Birder surpasses both the baseline and 1-bit Adam by a significant margin.

Figure 3 illustrates the system throughput of different optimizers when running ResNet-50 and BERT-Base on 8 GPUs to 64 GPUs (1 node to 8 nodes). When training on 8 GPUs (1 node), where computation takes precedence over communication, the throughput of Birder is slightly lower than that of SGDM and BertAdam . However, as the number of GPUs increases, Birder consistently outperforms its counterparts, and this superiority becomes increasingly evident with more GPUs. Additionally, the system throughput for SGDM , BertAdam , and 1-bit Adam occasionally decreases as the number of GPUs increases, whereas the throughput of Birder steadily grows. This observation indicates that Birder offers better scalability efficiency.

SGDM

25000

SGDM

BertAdam

15000 -

1-bit Adam

Birder

1-bit Adam

Birder

1000-

Birder

1-bit Adam

20000

800-

samples

10000

samples

15000

samples

600

10000

400 -

5000 -

5000

200

0

8 GPUs

16 GPUs

32 GPUs

64 GPUs

0

8 GPUs

16 GPUs

32 GPUs

64 GPUs

0

8GPUS

16 GPUs

32 GPUs

64 GPUS

(a) ResNet-50,

32

samples / GPU

(b) ResNet-50,

128

samples / GPU

(c) BERT-Base, 3 samples / GPU

SGDM Computation Time

SGDM Computation Time

1600 -

BertAdam Computation Time

SGDM Commuc Time

800 -

SGDM Communication Time

BertAdam Communication Time

400 -

1-bit Adam Computation Time

1-bit Adam Computation Time

1-bit Adam Communication Time

1-bit Adam Communication Time

1-bit Adam Computation Time

0000

1-bit Adam Comp./Decomp Time

1-bit Adam Comp./Decomp Time

1200 -

1-bit Adam Comp./Decomp Time

Birder Computation Time

600 -

Birder Computation Time

Birder Computation Time

ms / iter

300

Birder Comp./Decomp Time

/

Birder Comp./Decomp Time

Birder Communication Time

ms / iter

Birder Communication Time

Birder Comp./Decomp Time

400

800

200 -

100 -

200

400

0 -

8 GPUs

16 GPUs

32 GPUs

64 GPUs

8 GPUS

16 GPUs

32 GPUs

64 GPUs

0 -

8 GPUs

16 GPUs

32 GPUs

64 GPUs

(a) ResNet-50,

32

samples / GPU

(b) ResNet-50,

128

samples / GPU

(c) BERT-Base, 3 samples / GPU

In terms of inference performance for ResNet-50, we evaluate the Top-1 accuracy after training on ILSVRC2012 from scratch. For BERT-Base, we measure the F1-score and exact-match score after fine-tuning on SQuAD 1.1. Table 1 shows that when the batch size is set to 32 samples per GPU, the accuracy of Birder is slightly lower than that of SGDM . It has been suggested in some works ([13], [35]) that adaptive optimizers generally yield worse generalization compared to SGDM for CNN architectures. However, as the batch size increases (Table 2), both 1-bit Adam and Birder achieve better accuracy. This can be attributed to the beneficial effect of introducing a certain level of noise for generalization ([25]), which biases the optimizer towards wider valleys. Table 2 demonstrates that Birder achieves similar or higher F1-score and exact-match score compared to BertAdam and 1-bit Adam , validating the effectiveness of Birder for inference tasks.

5.3 Communication Efficiency Analysis

As shown in Figure 4, training on a single node demonstrates that the baseline SGDM and BertAdam algorithms are slightly faster compared to Birder and 1-bit Adam . In this scenario, the inter-GPU bandwidth within a node is extremely high, rendering communication time negligible. However, the newly introduced compression/decompression process by Birder and 1-bit Adam adds extra time due to its implementation. Thanks to light-computation quantization, the compression/decompression time for Birder with ResNet-50 and BERT-Base is significantly reduced to approximately 15 ms and 8 ms respectively.When conducting distributed training across two nodes, the bandwidth between them is relatively limited (10Gbps in our experiment), making communication time a critical factor. In the case of uncompressed SGDM and BertAdam , the communication time substantially exceeds the computation time for ResNet with 32 samples per GPU and BERT-Base. Consequently, the system throughput is lower compared to a single node (as depicted in Figure 2). However, the extreme 1-bit quantization implemented in Birder effectively reduces communication overhead, resulting in only a marginal increase in the total time required for Birder . As the number of nodes continues to increase, the importance of an efficient communication scheme becomes paramount. By leveraging

our proposed Hierarchical-1-bit-All-Reduce , the overall inter-node communication volume exchanged scales proportionally with the number of nodes. In contrast, the Compressed-All-Reduce method employed by 1-bit Adam [28] results in the overall communication volume exchanged among nodes being proportional to the number of GPUs (eight times larger than the number of nodes in our experiments). Consequently, as the number of nodes increases, the communication time for Birder exhibits a gradual rise, while the communication time for 1-bit Adam experiences a sudden surge.

6 Conclusion

In this study, we introduce a novel 1-bit adaptive optimizer for distributed training. Our optimizer offers the advantages of being lightweight in terms of computation while employing extreme 1-bit quantization for the communication data. Furthermore, we provide theoretical evidence demonstrating that Birder can achieve convergence rates comparable to the uncompressed Adam . To enhance communication speed, we propose a novel communication scheme tailored specifically for Birder , replacing the inefficient naive All-Gather approach. Through extensive experiments on benchmark models such as ResNet-50 and BERT-Base, we validate the effectiveness and efficiency of Birder in comparison to uncompressed methods like SGDM/Adam as well as the relevant 1-bit Adam .

Acknowledgments

This work is supported by the National R&D Program of China (Grant No. 2022ZD0115301), the Major Key Project of PCL (Grant No. PCL2023AS7-1), and the National Natural Science Foundation of China (Grant No. 61806128).

References

Saurabh Agarwal, Hongyi Wang, Shivaram Venkataraman, and Dimitris Papailiopoulos. On the utility of gradient compression in distributed training systems. In Proceedings of Machine Learning and Systems , volume 4, pages 652-672, 2022.

Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. arXiv preprint arXiv:1704.05021 , 2017.

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communicationefficient sgd via gradient quantization and encoding. In Advances in neural information processing systems , volume 30, 2017.

Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. Signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning , pages 560-569, 2018.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877-1901, 2020.

Congliang Chen, Li Shen, Haozhi Huang, and Wei Liu. Quantized adam with error feedback. ACM Transactions on Intelligent Systems and Technology (TIST) , 2021.

Jacques Chen, Frederik Kunstner, and Mark Schmidt. Heavy-tailed noise does not explain the gap between sgd and adam on transformers. In 13th Annual Workshop on Optimization for Machine Learning , 2021.

Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization. arXiv preprint arXiv:1808.02941 , 2018.

Minsik Cho, Vinod Muthusamy, Brad Nemanich, and Ruchir Puri. Gradzip: Gradient compression using alternating matrix factorization for large-scale deep learning. In Advances in Neural Information Processing Systems , 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Eduard Gorbunov, Konstantin P Burlachenko, Zhize Li, and Peter Richtárik. Marina: Faster non-convex distributed learning with compression. In International Conference on Machine Learning , 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. arXiv preprint arXiv:1712.07628 , 2017.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, and Yuxiong He. 1-bit lamb: Communication efficient large-scale large-batch training with lamb's convergence speed. arXiv preprint arXiv:2104.06069 , 2021.

Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08249 , 2020.

Yucheng Lu, Conglong Li, Minjia Zhang, Christopher De Sa, and Yuxiong He. Maximizing communication efficiency for large-scale training via 0/1 adam. arXiv preprint arXiv:2202.06009 , 2022.

Konstantin Mishchenko, Eduard Gorbunov, Martin Takáˇ c, and Peter Richtárik. Distributed learning with compressed gradient differences. arXiv preprint arXiv:1901.09269 , 2019.

Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations , 2018.

Peter Richtárik, Igor Sokolov, and Ilyas Fatkhullin. Ef21: A new, simpler, theoretically better, and practically faster error feedback. In Advances in Neural Information Processing Systems , 2021.

Mher Safaryan and Peter Richtárik. Stochastic sign descent methods: New algorithms and better theory. In International Conference on Machine Learning , 2021.

Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Fifteenth annual conference of the international speech communication association , 2014.

Shaohuai Shi, Xianhao Zhou, Shutao Song, Xingyao Wang, Zilin Zhu, Xue Huang, Xinan Jiang, Feihu Zhou, Zhenyu Guo, Liqiang Xie, et al. Towards scalable distributed training of deep learning on public cloud clusters. In Proceedings of Machine Learning and Systems , volume 3, pages 401-412, 2021.

Samuel L Smith and Quoc V Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations (ICLR) , 2018.

Sebastian Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for sgd with delayed gradients and compressed communication. arXiv preprint arXiv:1909.05350 , 2019.

Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. In Advances in Neural Information Processing Systems , volume 31, 2018.

Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit adam: Communication efficient large-scale training with adam's convergence speed. In International Conference on Machine Learning , pages 10118-10129, 2021.

Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. In Advances in Neural Information Processing Systems , volume 32, 2019.

Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright. Atomo: Communication-efficient learning via atomic sparsification. In Advances in Neural Information Processing Systems , volume 31, 2018.

Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in neural information processing systems , volume 30, 2017.

Feijie Wu, Shiqi He, Song Guo, Zhihao Qu, Haozhao Wang, Weihua Zhuang, and Jie Zhang. Sign bit is enough: a learning synchronization framework for multi-hop all-reduce with ultimate compression. In Proceedings of the 59th ACM/IEEE Design Automation Conference , 2022.

Hang Xu, Chen-Yu Ho, Ahmed M Abdelmoniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatsenidis, Marco Canini, and Panos Kalnis. Compressed communication for distributed deep learning: Survey and quantitative evaluation. Technical report, 2020.

Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems , 2020.

Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems , 33:21285-21296, 2020.

Appendix

A Theoretical Analysis for Algorithm 1

In practice, we implement Birder in a non-parameter-server model to further reduce the communication overhead, but the data exchange is essentially equivalent to that in a parameter-server prototype. Hence, we provide the theoretical analysis for Birder in a parameter-server model as shown in Algorithm 1.

According to Algorithm 1, the update ¯ u t can be recursively formulated as

(13)

Denote

Hence, the updating rule can be summarized as

A.1 Auxiliary Lemmas

Lemma 1. Let u t = m t b t , the element-wise quantization function is defined in Eq.(5) can be reformulated as

We have e t = u t -Q ( u t ) , and then the following holds true

Proof. From Eq.(15), we know

and,

Hence,

Lemma 2. Let x 0 = x 1 and α 0 = α 1 in Algorithm 1, defining the sequence

Then the following equality will hold, i.e.,

Proof. For t = 1 , we have

where the second equality follows the updating rule in Eq.(14).

For t ≥ 2 , following the updating rule in Eq.(14), we have

We know x t +1 -x t + α t ( e t -e t +1 ) = (1 -β )( x t +1 + -α t ( e t +1 -δ t )) -(1 -β )( x t -α t e t ) + β ( x t +1 -x t + α t ( δ t + e t -e t +1 )) , so Eq. (24) can be rearranged as

Divided both sides by 1 -β , we obtain

Rearranging Eq. (26), we have

Define the sequence

We finally obtain

Recalling x 1 = x 0 and α 1 = α 0 , we have α 1 b 1 = α 0 b 0 . Then, combining Eq.(23) and Eq.(29), we obtain the conclusion.

A.2 Proof of Theorem 1

Proof. By the the gradient Lipschitz continuous in Assumption 2 and Lemma 2, we obtain

glyph[negationslash]

glyph[negationslash]

where the second equality holds due to E [ δ t -1 ] = 0 and E [ e t -1 ] = 0 . The last inequality holds owing to E [ ‖ a + b ‖ 2 ] = E [ ‖ a ‖ 2 ] + E [ ‖ b ‖ 2 ] if E [ a ] = 0 or E [ b ] = 0 , and E [ ‖ a + b ‖ 2 ] ≤ 2 E [ ‖ a ‖ 2 ] + 2 E [ ‖ b ‖ 2 ] if E [ a ] = 0 and E [ b ] = 0 .

Taking telescope sum from 1 to T on the both sides of Eq.(30) , we then have

Now we focus on bounding T 1 below. From Assumption 4, we know ‖ g t ‖ ≤ G ( t = 1 , 2 , ..., T ) and ‖∇ f ( z t ) ‖ ≤ G . Due to m t = βm t -1 +(1 -β ) g t and m 1 = g 1 , it is easy to obtain ‖ m t ‖ ≤ G by complete induction.

Since ‖∇ f ( z t ) ‖ ≤ G and ‖ m t ‖ ≤ G , we have

where ( i ) holds sice ‖ a glyph[circledot] b ‖ ≤ ‖ a ‖ max j | ( b ) j | ≤ ‖ a ‖‖ b ‖ 1 , ( ii ) holds due to ‖∇ f ( z t ) ‖ ≤ G and ‖ m t ‖ ≤ G , ( iii ) holds because α t -1 ( b t -1 ) j -α t ( b t ) j ≥ 0 for any j ∈ [1 , 2 , ..., d ] , ( iv ) holds due to min j ( b t ) j ≥ ρ > 0 for any j ∈ [1 , 2 , ..., d ] .

Let us turn to bound T 2 ,

We now analyze T 5 below,

(34) where ( i ) holds by following 〈 a, b 〉 ≤ 1 2 ‖ a ‖ 2 + 1 2 ‖ a ‖ 2 , ( ii ) holds due to Assumption 1, ( iii ) holds due to Assumption 1 owing to Eq.(21), ( iii ) holds since E [ ‖ a + b ‖ 2 ] = E [ ‖ a ‖ 2 ] + E [ ‖ b ‖ 2 ] if E [ a ] = 0 or E [ b ] = 0 , ( v ) holds resulting from the updating rule in Eq. (14), ( vi ) holds due to ∣ ∣ ∣ ( m t ) j ( b t ) j ∣ ∣ ∣ ≤ 1 , | ( δ ) j | ≤ 2 (the definition of δ t in Eq. (11) ), E [ ‖ e t ‖ 2 ] ≤ d in Lemma 1, ‖ g t ‖ ≤ G in Assumption 2 and min j ( b t ) j ≥ ρ > 0 .

We then bound T 6

(35)

where ( i ) holds due to max j ( b t ) j ≤ ‖ b t ‖ ≤ G , ( ii ) holds owing to E [ ∇ f ( x t ) -g t ] = 0 in Assumption 2 and g t , b t -1 are independent, ( iii ) holds sice ‖ a glyph[circledot] b ‖ ≤ ‖ a ‖ max j | ( b ) j | ≤ ‖ a ‖‖ b ‖ 1 , ( iv ) holds resulting from ‖∇ f ( x t ) ‖ ≤ G and ‖∇ f ( x t ) -g t ‖ ≤ ‖∇ f ( x t ) ‖ + ‖ g t ‖ ≤ 2 G , and ( v ) holds because α t -1 ( b t -1 ) j -α t ( b t ) j ≥ 0 for any j ∈ [1 , 2 , ..., d ] , ( vi ) holds due to min j ( b t ) j ≥ ρ > 0 for any j ∈ [1 , 2 , ..., d ] .

Then, we pay attention to T 3 ,

where ( i ) holds due to ‖ a glyph[circledot] b ‖ ≤ ‖ a ‖‖ b ‖ , ( ii ) holds owing to ‖ m t -1 ‖ ≤ G , ( ii ) holds due to ‖ a ‖ 2 ≤ max j | ( a ) j |‖ a ‖ 1 , ( iv ) holds due to α t -1 ( b t -1 ) j -α t ( b t ) j ≥ 0 and α t ( b t ) j > 0 for any j ∈ [1 , 2 , ..., d ] , ( v ) holds resulting from min j ( b t ) j ≥ ρ > 0 for any j and α t is non-increasing, ( vi ) holds resulting from α t -1 ( b t -1 ) j -α t ( b t ) j ≥ 0 for any j ∈ [1 , 2 , ..., d ] , ( vii ) holds due to telescoping sum, and ( viii ) holds due to min j ( b t ) j ≥ ρ > 0 for any j ∈ [1 , 2 , ..., d ] ..

Now we turn attention to T 4 ,

where the inequality holds owing to ‖ m t -1 ‖ ≤ G and min j ( b t ) j ≥ ρ > 0 , ‖ ( δ t -1 ) j ‖ ≤ 2 , and E [ ‖ e t ‖ 2 ] ≤ d . Combining Eq.(31-37), we can obtain

Reformulating Eq.(38), we then have

It is known the learning rate saftifies α t = c √ t , ∀ t ≥ 1 and α 0 = α 1 = c . Utilizing non-increasing α t and Cauchy-Schwarz inequality, we know E [ ∑ T t =1 α t ‖∇ f ( x t ) ‖ 2 ] ≥ Tα T E [ 1 T ∑ T t =1 ‖∇ f ( x t ) ‖ ] 2 = √ T c E [ 1 T ∑ T t =1 ‖∇ f ( x t ) ‖ ] 2 . ∑ T t =1 α 2 t -1 = ∑ T t =1 c 2 t ≤ c 2 (1 + ∫ T -1 1 1 t dt ) ≤ c 2 (1 + log T ) , and ∑ T t =1 ( α t -1 -α t ) 2 = ∑ T t =2 ( α t -1 -α t ) 2 ≤ ∑ T t =2 c 2 4( t -1) 3 ≤ c 2 4 (1+ ∫ T -2 1 t -3 dt ) = c 2 4 ( 3 2 -1 2( T -2) ) ≤ 3 c 2 8 , we further have

where we define

B Experiments for Comparing Vanilla SGD, SGDM, Adam, Birder and SoftSignSGD

To address the bottleneck in communication during distributed training, numerous gradient compression algorithms have been proposed, aiming to reduce the communication volume. Most of these algorithms can be reduced to Vanilla SGD without momentum if compression is not performed. Generally speaking, the epoch-wise convergence rate and inference performance a compressed algorithms is upper bounded by its uncompressed counterpart. In the experiments, we conducted empirical experiments to evaluate the training and inference performance of of Vanilla SGD , SGDM , Adam , Birder and its uncompressed version in training typical CNN-base, LSTM-base and Transformer-base DNNs.

We refer to the uncompressed Birder as SoftSignSGD . The implementation details for SoftSignSGD are presented in Algorithm 1. When comparing SoftSignSGD to Adam , there are two key differences. First, instead of using the square root of the exponential moving average of the squared gradient, denoted as √ v t = √ (1 -β 2 ) v t -1 +(1 -β 2 ) g 2 t , SoftSignSGD utilizes the exponential moving average of the absolute gradient, represented as b t = (1 -β ) b t -1 + | g t | . Second, in SoftSignSGD , the exponential moving factors for both the numerator m t and the denominator b t are the same. These differences ensure that each element of the updating amount in SoftSignSGD satisfies the condition -1 ≤ ( m t b t ) j ≤ 1 .

B.1 Experimental Results for training ResNet-20

We evaluated the performance of five optimization algorithms: Vanilla SGD , SGDM , AdamW , Birder and SoftSignSGD , for training ResNet-20 on CIFAR100. Each batch consisted of a set of 128 examples sampled with replacement. For SGDM , we set the momentum parameter β to 0 . 9 , while for SoftSignSGD and Birder , it was set to 0 . 95 . For AdamW , the parameters β 1 and β 2 were set to 0 . 9 and 0 . 999 , respectively. The weight decay was uniformly set to 0 . 0005 for Vanilla SGD and SGDM , and 0 . 05 for AdamW , Birder and SoftSignSGD . To simplify the tuning process and ensure fair comparisons, we initialized the learning rates at 0 . 1 for Vanilla SGD and SGDM , and 0 . 005 for AdamW , Birder and SoftSignSGD . We divided the learning rates by 10 after 75 and 130 epochs, and terminated the training after 150 epochs.

Figure 5 visually demonstrates that Vanilla SGD exhibits slower convergence speed and lower test accuracy compared to SGDM . In contrast, both Birder and SoftSignSGD show comparable training and inference performance to the commonly used SGDM and AdamW . This observation suggests that Birder outperforms existing gradient compression algorithms when training CNN-based ResNet-20 models.

(a) Train Loss

(b) Test Accuracy

VanillaSGD

SGDM

60

Adamw

SoftSignSGD

3

Birder

Test Accuracy

Train Loss

40

2

20

SGDM

Adamw

SoftSignSGD

Birder

0

30

60

90

120

150

0

30

60

90

120

150

Epoch

Epoch

B.2 Experimental Results for training LSTM

We conducted experiments to train a 3-layer LSTM model on the Penn TreeBank dataset to evaluate the performance of five optimization algorithms: Vanilla SGD , SGDM , AdamW , Birder and SoftSignSGD . Our implementations were built upon the code provided in the AdaBelief paper 7 , and we used the default experimental settings for SGDM and AdamW . For Vanilla SGD , we used the experimental settings of SGDM with the exception that we set the momentum parameter β to 0. For Birder and SoftSignSGD , we adopted the experimental settings of AdamW , except that we set the momentum parameter β to 0 . 99 .

As visually illustrated in Figure 6, Vanilla SGD is still less effective than SGDM in terms of the convergence speed and the test accuracy, while the training and inference performance of SoftSignSGD and Birder are comparative to common-used SGDM and AdamW . It indicates the Birder is superior to exiting gradient compression algorithms for training LSTM.

B.3 Experimental Results for training ViT

We train ViT-B with Vanilla SGD , SGDM , AdamW , SoftSignSGD and Birder on the ILSVRC2012 with 32 GPUs (4 nodes). We use the Pytorch official implementation for ViT 8 . For AdamW , SoftSignSGD and Birder , we followed the recommended experimental settings, with the exception that we set the momentum parameter β to 0.95 for SoftSignSGD and Birder . As for Vanilla SGD and SGDM , we set the basic learning rate to 0 . 1 and the weight decay to 0 . 001 , while keeping other settings the same as AdamW . Instead of the default 300 epochs, we uniformly set the total number of epochs to 150 for all optimizers

As visually illustrated in Figure 7, Vanilla SGD is still less effective than SGDM in terms of the convergence speed and the test accuracy, while the training and inference performance of SoftSignSGD and Birder are comparative to common-used SGDM and AdamW . Notably, the performance of SGD-type optimizers are substantially inferior to that of adaptive optimizers.

(a) Train Loss

(b) Test Perplexity

150

VanillaSGD

SGDM

SGDM

Adamw

SoftSignSGD

Adamw

SoftSignSGD

5

Birder

Test Perplexity

Birder

Train Loss

11

100

4

3

50 -

0

50

100

150

200

0

50

100

150

200

Epoch

Epoch

(a) Train Loss

(b) Test Accuracy

VanillaSGD

80

SGDM

Adamw

SoftSignSGD

60

6

Birder

Loss

Train

40

Test

VanillaSGD

4

SGDM

20

Adamw

SoftSignSGD

Birder

2

0

30

60

90

120

150

30

60

90

120

150

Epoch

Epoch

C Experimental Results for pre-training BERT-Base

We employed BertAdam and Birder to pre-train BERT-Base on Wikipedia using 64 GPUs ( 8 nodes). The sequence length was set to 512 , and the batch size per GPU was set to 16 . The training process consisted of 37 , 000 iterations. The learning rate started at 4 × 10 -4 and linearly increased in the first 12 , 500 iterations, after which it linearly decreased to 0 for the remaining iterations. For BertAdam , the parameter values [ β 1 , β 2 ] were set to [0 . 9 , 0 . 999] , and for Birder , the momentum parameter beta was set to 0 . 9 .

As depicted in Figure 8, Birder demonstrates a comparable iteration-wise convergence rate to BertAdam . However, in terms of time-wise convergence, Birder achieves a 4.2x faster convergence rate compared to BertAdam .

D Experiments with InfiniBand connections

To further evaluate the communication efficiency of SGDM/Adam , SoftSignSGD and Birder with high bandwidth connections, we implement experiments for training ResNet-50 and BERT-Base with distributed nodes connected with 200Gbps InfiniBand. All the experimental settings are the same as we perform experiments with Ethernet in Subsection 5.1, and the experimental results are listed in Table 3 and Table 4.

As shown in Table 3 and Table 4, compared with the baseline SGDM/Adam , Birder can still reach up to 1 . 45 × speedup for ResNet-50 on ILSVRC2012 and 2 . 85 × speedup for BERT-Base on SQuAD 1.1, although the speed advantage is not so obvious as that with lower-bandwidth Ethernet connections. An interesting phenomenon

(a) Iteration-wise, BERT-Base, batch size= 16 × 64

(b) Time-wise, BERT-Base, batch size= 16 × 64

BertAdam

BertAdam

Birder

Birder

b

SS07

SS07

Q

5K

10K

15K

20K

25K

30K

35K

10

15

20

25

30

35

Iteration

Wall clock time (hour)

is that the system throughput of Birder with 10Gbps Ethernet can match that of SGDM/Adam with 200Gbps InfiniBand.

The experimental results in Table 3 and Table 4 also show that as the number of GPUs is increasing, the scale efficiency of SGDM/Adam , SoftSignSGD and Birder becomes lower. The reason for this phenomenon can be summarized in the following. When the number of GPUs doubles, the number of communication trips also

multiplies. We take the communication scheme All-Reduce for example. If the number of GPUs is n , each GPU requires 2( n -1) trips across the network confections. When the number is non-trivial, the computation time of the communication primitives may exceed the time of the pure communication itself and dominate the overall communication time, since the total communication overhead does not change with the number of GPUs. Notably, All-reduce is more efficient than All-to-All which is the core of our Hierarchical-1-bit-AllReduce . Hence, as shown in in Table 3 and Table 4, the scale efficiency of Birder decreases more quickly than SGDM/Adam with the number of GPUs growing.

E Discussion

The original paper on 1-bit Adam reports a significant speed advantage (up to 3 . 8 × ) for 1-bit Adam compared to full-precision Adam , with the advantage becoming more prominent as the number of GPUs increases. However, in our experiments, we did not observe clear speed advantages for 1-bit Adam over the original Adam. In fact, when running on 64 GPUs, 1-bit Adam was not only slower than the original Adam, but its throughput rate was even lower than that on 32 GPUs. There are several reasons for this phenomenon. First, the speedup of 1-bit Adam is obtained by comparing the throughput of the compression phase with that of the warm-up phase. However, in our experiments, we evaluated the overall average throughput of both the warm-up phase and the compression phase for 1-bit Adam . Second, the baseline Adam did not run with system-level efficient DDP . Third, the authors of 1-bit Adam customized highly efficient communication primitives specifically for their optimizer, whereas we utilized off-the-shelf communication primitives in PyTorch for all the optimizers to ensure fairness.

As shown in Figure 4, as the number of GPUs increases, the communication time for Birder also grows superlinearly. One of the reasons for this is that the communication primitive All-to-All accounts for an increasing portion of the communication time. However, the native All-to-All in Step (iii) of the Hierarchical1-bit-All-Reduce is not less efficient than the native All-Reduce . Therefore, we plan to further optimize the All-to-All and All-Gather primitives to accelerate Birder .

When training large-scale DNNs, the mixed-precision technique is commonly used to reduce memory consumption, allowing for larger model sizes. While optimizers still utilize full-precision states and computations, which typically contribute to 33-75% of the total memory footprint, Birder does not require full-precision states or computations. Moreover, due to the random quantization of updates to 1 or -1, Birder can leverage lower precision than FP16 gradients to estimate the update. Therefore, Birder shows promise for applications that focus on reducing memory usage, as highlighted in recent research on 8-bit optimizers via block-wise quantization (Tim Dettmers et al., ICLR 2022).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWith the increasing size of the model, communication compression plays an important role in distributed machine learning because the model needs to transmit among the nodes. Toward this goal, this paper introduces a one-bit adaptive optimizer named BRAM and further extends the method to hierarchical all-reduce. Specifically, the method is lightly computed because it requires compression only, which significantly saves the time spent on the decompression step. Theoretical analysis shows the efficiency of the proposed algorithm, and the empirical studies validate the method is comparable with one-bit Adam and uncompressed SGDM.\n\nSTRENGTHS:\n**S1.** The authors give a convergence rate of the proposed algorithm, which matches the state-of-the-art approach. Although I do not check the proof details, the result sounds reasonable. \n\n**S2.** The experiment setup is comprehensive. It evaluates ResNet-50 and BERT-based models and conducts experiments on various numbers of GPUs.\n\nWEAKNESSES:\n**W1.** The literature review is poor. The authors did not comprehensively survey the works of one-bit compression, e.g., SSDM [1] and Marsit [2]. The unbiased design of Equation (5) has previously been proposed by the method proposed in [1], and [2] discusses a more complex all-reduce case. \n\n**W2.** This work is not very surprising. The existing works on one-bit compression generally use Adam/Momentum as the adapters in their experimental evaluation [1, 2, 3], although their approaches are discussed under SGD settings. \n\n**W3.** As the paper is discussed under the one-bit transmission, the readers expect the proposed method can use single-bit only for each parameter at every transmission. However, the proposed method seems incapable under multihop all-reduce [2] settings. \n\n***\n\n**References:**\n\n[1] Safaryan, Mher, and Peter Richtárik. \"Stochastic sign descent methods: New algorithms and better theory.\" International Conference on Machine Learning. PMLR, 2021. \n\n[2] Wu, Feijie, et al. \"Sign bit is enough: a learning synchronization framework for multi-hop all-reduce with ultimate compression.\" Proceedings of the 59th ACM/IEEE Design Automation Conference. 2022. \n\n[3] Bernstein, Jeremy, et al. \"signSGD with Majority Vote is Communication Efficient and Fault Tolerant.\" International Conference on Learning Representations. 2018.\n\nQUESTIONS:\n**Q1.** The introduction claims that the proposed algorithm performs similarly to uncompressed SGDM/Adam. However, the experiments compare the proposed work with one-bit Adam only. Is it over-claimed because we cannot see how close it is between the proposed work and uncompressed Adam? Can the authors provide the results for uncompressed Adam?\n\nLIMITATIONS:\n**L1.** The method designed for Hierarchical-1-bit All-Reduce cannot guarantee every transmission using 1 bit only. It will require extra steps to encode the transmitted gradients at the sender and decode them at the receiver.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
The authors propose a novel 1-bit adaptive optimizer by compressing the total update in each iteration. Besides, the author modifies the adam update with the absolute value of the gradient. No bias correction is needed and error feedback is applied in algorithm 1. Hierarchical-1-bit-All-Reduce is proposed, where intra-node communication is full precision, while inter-node communication is compressed. Experiments verifies that the proposed method achieves faster convergence and the model performance is roughly the same.

STRENGTHS:
- experiments show wall-clock time acceleration under different local batch size settings. Typically, when local batch size is larger, the computation to communication ratio is higher, and communication compression will lead to lower speedup compared with full-precision training.
- Hierarchical-1-bit-All-Reduce is proposed and sounds reasonable, since the intra-node communication could have a higher bandwidth.
- the wall-clock training time comparison explicitly show the faster convergence speed of the proposed method.

WEAKNESSES:
- all-gather instead of all-reduced is used in figure 1
- noticeable performance drop could be observed from Table 1
- gradient norm needs to be bounded in assumption 4
- the theoretical analysis is rather short and lacks insight.

QUESTIONS:
- m and b are not synchronized across workers. how do you make sure that the workers do not diverge because of different b?
- have you tried signsgd instead of Eq(5)? Error feedback could work better for biased compressor.
- how do you get rid of batch size in convergence results, which is considered as a theoretical difficulty in Adam?

LIMITATIONS:
The authors did no discuss societal impact. I don't see potential negative societal impact neither.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a compute-efficient gradient compressor named BRAM which randomly quantifies the update direction into 1-bit for communication. A theoretical convergence bound is also provided to prove BRAM’s convergence property. To improve communication efficiency, BRAM also integrates a novel Hierarchical-1-bit-All-Reduce algorithm that can better utilize high-bandwidth intra-node communication. Experiments conducted on two popular DNNs (ResNet-50 with ImageNet and BERT-Base with SQuAD1.1) show BRAM seems to outperform SGDm, Adam, and 1-bit Adam.\n\nSTRENGTHS:\n1. A compute-efficient lightweight compressor on model updates (not gradients) is proposed to reduce communication volume in distributed data-parallel training.\n2. A novel *Hierarchical-1-bit-All-Reduce* algorithm is proposed to better utilize the high-bandwidth intra-node communication and alleviate the communication through low-bandwidth inter-node interconnect.\n\nWEAKNESSES:\n1. Experiments are not strong to support the claims of the paper. \n2. Some state-of-the-art compressed methods like HiTopKComm (with MSTop-k) and PowerSGD were not compared.\n\nQUESTIONS:\n1. Experiments seem not to be conducted fairly. 1) The ResNet-50 model was not trained with standard accuracy (i.e., 75.9% top-1 accuracy according to MLPerf and many other benchmarks). For example, in Table 1, top-1 accuracy drops to 74.+% on 64 GPUs with 128 samples per GPU (i.e., a global batch size of 8K), but many studies show ResNet-50 on ImageNet can be scaled to a mini-batch size of 32K to reserve 75.9% top-1 accuracy. 2) The testbed has a very high intra-node bandwidth but relatively low inter-node bandwidth, thus the *Hierarchical-1-bit-All-Reduce* algorithm is important. However, the experimental setting seems deliberately constructed to show the effectiveness of BRAM according to Table 1 where only 128 samples were used for each Nvidia A100 GPU such that the computation-to-communication ratio is low. To fairly compare with SGDM, the local batch size is possible to be 512 or 1024 on an Nvidia A100-80G GPU. It is similar to the BERT-Base experiments, where the local mini-batch size is only 3. Is it out of memory if setting a larger mini-batch size?\n\n2. The paper claims that BRAM doesn’t need a warmup strategy to align the accuracy to the dense version, but the benchmarks don’t have standard accuracy. Is it possible that 1-bit Adam, MSTop-k (with HiTopKComm), and PowerSGD also achieve similar accuracy achieved by BRAM even if they train DNNs without warmup? How about comparing MSTop-k and PowerSGD in terms of end-to-end training time to achieve target accuracy?\n\n3. It seems that the convergence analysis in Section 3 is based on the theoretical framework from [8]. Is the theoretical convergence analysis providing new insights in this paper compared with [8]?\n\n4. The *1-bit-All-Reduce* algorithm converts all-reduce to all-to-all, how about the communication complexity of *Hierarchical-1-bit-All-Reduce* compared to state-of-the-art all-reduce algorithms (e.g., ring-based, tree-based, etc.)? A communication complexity comparison is suggested to be provided to help understand the advantage of  Hierarchical-1-bit-All-Reduce.\n\n5. The key idea of the *Hierarchical-1-bit-All-Reduce algorithm* is almost the same with HiTopKComm in [23] though they are used under different scenarios. Should credit be given to [23]? and what are their main differences?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a 1-bit adaptive optimizer called BRAM for distributed training of deep neural network models. BRAM combines the merits of SignSGD and Adam optimizers to achieve fast convergence speed, high inference performance, and reduced communication overhead. The proposed optimizer employs extreme 1-bit quantization for communication data while being lightweight in terms of computation. A hierarchical communication scheme called Hierarchical-1-bit-All-Reduce is also introduced to enhance communication efficiency. Extensive experiments on benchmark models, including ResNet-50 and BERT-Base, validate the effectiveness and efficiency of BRAM compared to uncompressed methods like SGDM/Adam and the 1-bit Adam optimizer. The experimental results demonstrate that BRAM achieves similar convergence rates to the baseline methods while significantly outperforming them in terms of training speed and system throughput, especially with a larger number of GPUs. Inference performance results show that BRAM performs comparably or even better than the baseline methods. Overall, BRAM offers a promising solution for distributed training with improved communication efficiency and convergence speed.\n\nSTRENGTHS:\n- The paper introduces BRAM, a 1-bit adaptive optimizer that combines the strengths of SignSGD and Adam. This optimizer effectively reduces communication overhead while maintaining fast convergence speed and high inference performance.\n\n- The paper provides a theoretical convergence analysis for BRAM under certain assumptions. The analysis demonstrates that BRAM achieves a convergence rate comparable to uncompressed Adam, providing a solid theoretical foundation for its effectiveness.\n\n- It proposes a hierarchical communication scheme called Hierarchical-1-bit-All-Reduce specifically designed for BRAM. This scheme leverages high intra-node bandwidth and reduces inter-node communication overhead, resulting in better scalability efficiency and improved system throughput.\n\n- Extensive experimental validation: The paper conducts extensive experiments on benchmark models, including ResNet-50 and BERT-Base, LSTM, ViT etc., using different numbers of GPUs. The experimental results demonstrate the superiority of BRAM over uncompressed methods like SGDM/Adam and the 1-bit Adam optimizer in terms of training speed and system throughput, while maintaining comparable or even better inference performance.\n\nWEAKNESSES:\n- The theoretical convergence analysis of BRAM relies on several assumptions, such as bounded infimum, Lipschitz continuous gradient, and unbiased and independent noisy gradients. The sensitivity of BRAM's performance to these assumptions should be discussed to provide a more realistic assessment of its convergence properties.\n\n- The paper briefly mentions the selection of hyperparameters such as the learning rate and exponential moving average factor, but there is limited discussion on the sensitivity of BRAM to these hyperparameters. A more thorough analysis and discussion of hyperparameter tuning would provide insights into achieving optimal performance. The paper does not include ablation studies to analyze the individual contributions of different components of BRAM. Investigating the impact of each component and analyzing their relative importance would enhance the understanding of the optimizer and provide insights for further improvements.\n\n- Limited analysis of robustness to noisy gradients: The paper does not extensively discuss the robustness of BRAM to noisy gradients. While the theoretical analysis assumes unbiased and independent noisy gradients, a more in-depth investigation of BRAM's performance in the presence of varying levels of gradient noise would provide a comprehensive understanding of its behavior in practical scenarios.\n\n- Although the paper mentions the reduced compression/decompression time of BRAM compared to 1-bit Adam, a more detailed analysis and comparison of the time required for compression and decompression would provide a better understanding of the overhead introduced by the quantization process.\n\n- The paper focuses on communication efficiency but does not thoroughly discuss the memory efficiency of BRAM. Analyzing the memory requirements of BRAM compared to other optimizers would be valuable, especially in scenarios with limited memory resources.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nThere is no limitation discussion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

comments: SUMMARY:
The paper proposes BRAM, a new 1-bit optimizer that directly quantizes the entire adaptive update. It theoretically proves that BRAM promises the same convergence speed as the full-precision Adam even with extreme 1-bit quantization. Considering the difference between intra-node and inter-node bandwidth, the paper also proposes Hierarchical-1-bit-All-Reduce,  a hierarchical communication scheme that aggregates our 1-bit data by leveraging the ultra-high intra-node bandwidth. The experiments show that BRAM can achieve empirical high performance on top of Hierarchical-1-bit-All-Reduce.

STRENGTHS:
1. The presentation is good. The writing is clear and easy to follow.
2. The paper makes a detailed evaluation and analysis of 1 Adam.
3. The appendix proves that the element-wise quantization function is unbiased.

WEAKNESSES:
1. It is unclear whether using the same hyperparameters for both the exponential moving averages of gradients and the squared gradients would have negative effects.
2. BRAM has less room for further optimization and can only be employed in very few applications because it sends updates directly back.

QUESTIONS:
Can BRAM be applied to pipeline parallel distributed training of foundation models?

LIMITATIONS:
The authors do not mention the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26