SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
Authors: Jinda Jia
Cong Xie
Hanlin Lu
Daoce Wang
Hao Feng
Chengming Zhang
Baixi Sun
Haibin Lin
Zhi Zhang
Xin Liu
Dingwen Tao
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM Training
Quantization
Communication Reduction
Collective Communication
Primary area: infrastructure
TL;DR: The first work that successfully reduces both gradients and weights to nearly 4 bits without compromising training accuracy in distributed training of large language models.

Abstract: Recent years have witnessed a clear trend towards language models with an ever-increasing number of parameters, as well as the growing training overhead and memory usage. Distributed training, particularly through Sharded Data Parallelism (ShardedDP) which partitions optimizer states among workers, has emerged as a crucial technique to mitigate training time and memory usage. Yet, a major challenge in the scalability of ShardedDP is the intensive communication of weights and gradients. While compression techniques can alleviate this issue, they often result in worse accuracy. Driven by this limitation, we propose SDP4Bit (Toward 4Bit Communication Quantization in Sharded Data Parallelism for LLM Training), which effectively reduces the communication of weights and gradients to nearly 4 bits via two novel techniques: quantization on weight differences, and two-level gradient smooth quantization. Furthermore, SDP4Bit presents an algorithm-system co-design with runtime optimization to minimize the computation overhead of compression. Additional to the theoretical guarantees of convergence, we empirically evaluate the accuracy of SDP4Bit on the pre-training of GPT models with up to 6.7 billion parameters, and the results demonstrate a negligible impact on training loss. Furthermore, speed experiments show that SDP4Bit achieves up to 4.08× speedup in end-to-end throughput on a scale of 128 GPUs.

SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
Jinda Jia" Indiana University jinjja@iu. edu
Cong Xie ByteDance Inc. cong wwwebyytedancee.ooooooo com
Hanlin Lu ByteDance Inc.
hanlin. wwwbyyedanceeccom com
Daoce Wang Indiana University daocwang@iu. edu
Hao Feng
Chengming Zhang
Indiana University haofenggiu. edu
University of Houston
czhang598cougarnet. uh. edu
Baixi Sun Indiana University sunbaix@iu. edu
Haibin Lin
Zhi Zhang
Byte Inc.
ByteDance Inc.
haibin. Srrriin.in@nytttne www@byteddnnee.com zhangzhi. hhangaii.johuuabbytedaacc... jowhua@byteaanceee..... com
Xin Liu ByteDance Inc. liuxin. ai@bytedance.com
Dingwen Tao Indiana University ditao@iu. edu
Abstract
Recent years have witnessed clear trend towards language models with ever- increasing number of parameters, as well as the growing training overhead and memory usage. Distributed training, particularly through Sharded Data Parallelism (ShardedDP) which partitions optimizer states among workers, has emerged as crucial technique to mitigate training time and memory usage. Yet, major challenge in the scalability of ShardedDP is the intensive communication of weights and gradients. While compression techniques can alleviate this issue. they often result in worse accuracy. Driven by this limitation, we propose SDP4Bit (Toward 4Bit Communication Quantization in Sharded Data Parallelism for L.M Training), which effectively reduces the communication of weights and gradients to nearly 4 bits via two novel techniques: quantization weight differences, and two- level gradient smooth quantization. Furthermore, SDP4Bit presents an algorithm- system co-design with runtime optimization minimize the computation overhead of compression. In addition to the theoretical guarantees of convergence, we empirically evaluate the accuracy SDP4Bit on the pre-training of GPT models with up to 6.7 billion parameters, and the results demonstrate negligible impact on training loss. Furthermore, speed experiments show that SDP4Bit achieves up to 4.08 x speedup in end-to-end throughput on scale of 128 GPUs.
Introduction
Large Language Models (LLMs) are increasingly utilized across various applications, leading to trend toward larger model sizes. This expansion in model size significantly escalates training overheads, making the process more costly and resource-intensive. To mitigate the time-consuming
"Equal Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

nature of training LLMs, it is common to employ multiple GPUs in data-paralled configuration. However, naive Data Parallelism (DP) necessitates that each GPU replicates the entire optimizer states, strategy often impractical due to the limited memory capacity of individual GPUs. This limitation becomes particularly critical with the substantial size of modern L.MSs.
Sharded Data Parallelism (ShardedDP) evolves from naive DP to reduce the memory foot- print by sharding optimizer states among GPUs. However, the sharding mechanism significantly changes the communication pattern of DP, which brings up new challenges in system op- timization. As a result, ShardedDP suffers from heavy communication overheads of both weights and gradients, particularly when inter- node bandwidth is limited. This can significantly increase the end-to-end (E2E) training time, es- pecially when using a small gradient accumula- tion step.
2.5
1.875
2.4
1.850
Validation Loss
2.3
1.825
2.2
1.800
72000
76000
80000
2.1
2.0
1.9
Baseline
SDP4Bit
1.8
ZeRO++
20000
40000 Iteration
60000
80000
Quantization a widely used strategy to reduce the communication overhead of naive DP, albeit with some accuracy loss. Unfortunately, few prior studies have specifically addressed the is- sue of communication reduction in ShardedDP. Recently, QSDP [18] and ZeRO++ [32] attempted to quantize the communication of ShardedDP to Int4. However, when pushing the communication ratio to its limits, both QSDP and ZeRO++ fail to maintain comparable training loss to the baseline. Furthermore, ZeRO++ lacks theoretical convergence guarantees, and QSDP is limited to one specific quantizer called "random shift" and strong assumptions. Thus, there no effective solution to reduce ShardedDP's communication to nearly 4 bits without compromising the training loss.
Figure 1:
Training validation loss for GPT-6.7B;
SDP4Bit is closely aligned with full precision training.
To address these issues, this paper proposes a novel communication reduction strategy, SDP4Bit. SDP4Bit comprises two main techniques: (1) Quantization on Weight Differences: Instead of directly quantizing weights, we apply 4-bit quantization to compress the weight differences between current and previous iterations; (2) Two-Level Gradient Smooth Quantization: We apply 8-bit quantization intra-node gradients and 4-bit quantization inter- gradients, with Hadamard Transform for smoothing the outliers. To the best of our knowledge, SDP4Bit is the first work to successfully reduce both gradients and weights to nearly 4 bits without compromising training accuracy. As shown in Figure 1, the training validation loss for GPT-6.7B using SDP4Bit is closely aligned with full precision training. Our main contributions are summarized as follows:
We propose low-bit (i.e., nearly 4-bit) communication reduction strategy for ShardedDP that preserves E2E training accuracy.
We establish convergence guarantee for the proposed strategy, showing the same convergence rate as the ordinary Stochastic Gradient Descent (SGD), with extended choices of biased compressors and weaker assumptions compared to the previous theoretical results.
We implement our method within the Megatron-LM framework and enhance it with runtime optimizations such buffer reuse, operation pruning, and kernel fusion.
Our results validate that SDP4Bit successfully compresses the communication of weights and gradients to nearly 4 bits, with negligible impact final loss. Notably, compared to non-quantized baseline, it achieves 4.08x speedup for a GPT-188 model trained on 128 H800 GPUs.
2 Preliminaries
2.1 Sharded Data Parallelism
Sharded Data Parallelism (ShardedDP) modifies traditional Data Parallelism (DP) to reduce the mem- ory footprint per GPU. Unlike traditional DP, which duplicates high-precision optimizer states (typ- ically including model weights and momentum variables in Float32) on each GPU, ShardedDP partitions them across all GPUs. Each GPU manages P of the optimizer states, hence reducing the corresponding memory footprint by factor of 3. where P represents the number of GPUs involved.
2

---

Algorithm QSDP/ +eRO++
Algorithm 2 Megatron-LLM with SDP4Bit
Require: worker: p. weight in shard p: w(p). local gradi Require:: worker: p. weight in shard pi w(p). local ent on worker Imodel global gradient in shard
gradient worker p: I global gradient in shard p: Imain (p), weight difference: d
Imoin
1: function CompreseedForwardPaas
1: function CompreseedForarrdPas
2: Emain po] f- QuantizeWeitn Weights Wmain (p])
dlp -mmein (p] - Wmodel (p)
3: Wmoded <- A mainn (p))
dlp) QuuntizeWeightsDifffiee
4:
output? <- ForwardPass imppul"
4: d + Al[Gather(d[p)
5:
mooee
Wmodel Wmodel +
6:
function CompressedBackwardPass
output" ForwardPass Wmodet imput")
7:
Umain [o] + QuantizeWeights/ (wer (imain (p])
7:
function CompressedBackwardPPas
8:
Umodel AllGather(( mainn
8: Pooddt Gradient( Wmodel oulput!?)
9:
Pmoste < Gradient( (mmdeii, output")
9: Imain p) f ReduceScatter HS (HHHS(())) mondet)
10:
free (W0dee))
10: Wmain P) Optimizer(gmaise (p), Umain P)
Fmodet
QuantizeGradients((ff (f
12:
ReduceScatter/ TwoAllo (l
13:
Wmain Optimizer(gmain jo], Wmasn [p])
With high-precision model weights sharded across GPUs (referred to as "main weights"), an all- gather operation required to collect the weights for the forward-backward steps (typically in relatively low precision, such as Float16, referred to as "model weights"). For gradient synchro- nization, reduce-scatter scatter operation performed before the optimization steps to ensure that each GPU has the corresponding shard of averaged gradients. In summary, each iteration necessitates an all all-gather for weights and a reduce-scatter for gradients.
Driven by the need to train larger models within the constraints of GPU memory, ShardedDP is incorporated into several popular training frameworks, including Megatron-LM (Distributed Optimizer), DeepSpeed (ZeRO), and PyTorch (FSDP), each with slightly different implementation strategies. Notably, ZeRO-3 and FSDP release the collected weights after each computation to enhance memory efficiency, necessitating additional weight collective communication during the backward pass. Conversely, ZeRO-2 and Megatron-LAN -MM retain the collected weights throughout, thus eliminating the need for weight collection during the backward pass. Our weight reduction strategy is particularly well-suited for Megatron-LM, as it maintains full model's weights at all times (see Algorithm 2). Additionally, Megatron-LMM, LM provides flexible parallelism support, such as tensor parallelism, which partitions models vertically to alleviate memory limitations. This approach enables the training of larger models compared to DeepSpeed.
2.2 Quantization
Quantization is commonly used strategy in data compression. In this paper, we explore symmetric linear (integer) quantization due to its low overhead and latency. It is defined as follows:
Time = round
(2*-1-1) 8 = max(x);
where & represents the bit-width of the quantized values, and s is referred to as "scales".
Additionally, group-wise quantization [25] is employed to minimize quantization error by dividing the data into multiple groups and quantizing each group individually. This approach results in a lower compression ratio due to the need to store additional scales.
2.3 Collective Reduction Communication with Quantization
State-of-the-art (SOTA) collective communication libraries (c.g., NCCL, Gloo) employ ring-based algorithm for its optimal bandwidth [21]. This algorithm executes reduce-scatter operations across P- 1 rounds, during which each GPU sends local data and aggregates the received data. When quantization is applied, this necessitates P - rounds of quantization and dequantization, potentially leading error propagation and increased latency [11]. Some strategies replace reduce-scatter with all-to-all communication, but this increases inter-node communication, typically with lower bandwidth.
ZeRO++ [32] modifies this approach by substituting the conventional reduce-scatter (used by QSDP) with two all-to-all operations (shown in Algorithm 1, with different colors to distinguish ZeRO+
3

---

from QSDP). The first operation is confined within each node, and post-reduction, the data size is diminished v. where . is the number GPUs per node. The subsequent all-to-all operation occurs between GPUs across different nodes that share the same local rank. In ZeRO++, each all-to-all operation follows 4-bit quantization step to mínimize communication data size.
As shown in Figure 5, while this method efficiently integrates quantization into reduce-scatter without augmenting inter-node communication, the repeated 4-bit quantization steps can accumulate quantization errors, potentially leading to suboptimal training outcomes.
Model Main Weights
Weight Differences
Dequantize
GPU
Qunnt
Node
Nodo
Sub
GPU
GPU
GPU
GPU
Sub
Nadea 2
Node
Quant
PPV
GPU
GPU
GPU
GPU1
Gune
GPU 2
GPU
GPU
GPU3
GPU
GPU
GPU2
GPU
Quantize
Step2:
Step3: Dequantize
Figure 3: Two-level gradient quantiza- tion: 8-bit intra-node and 4-bit inter- node quantization.
Weight
Weight Diff All Gather
Add Back
Figure Communication of quantized weight differences.
3 Methodology
3.1 Quantization on Weight Differences (qDD)
As discussed in Section 2.1, ShardedDP requires each GPU to send/receive up- dated weights (main weights) to/from other GPUs in each iteration. However, weights generally exhibit wide range and directly applying 4-bit quantization leads signifi- cant quantization errors. Even with group- wise quantization, gap E2E training loss compared to full precision remains de- spite using small group sizes.
Interval 1.9e-01
10"
Interval 1.1e-03
Prcaency (og scale
10
10
405
10F
100
10P
10P
10:
1.3c+00
1.30+00
7.86.03
WD: To address this issue, we quantize weight differences instead of the original weights during communication. As illus- trated in Figure 2 and Algorithm after the optimizer step, each GPU calculates the differences between the main weights and the model weights. These differences are then quantized and all all-gathered across all GPUs, After all all-gathering, each GPU dequantizes the received data to obtain the weight differences. These differences are then added the model weights obtain the updated weights.
Volue
value
Figure 4: Histogram of (a) weights and (b) weight differences. Each vertical dashed line represents quantization level sponding a 4-hit quantization lattice.
There main benefits applying quantization weight differences.
practice, weight differences are generally easier to quantize. As shown in Figure 4, weight differences are more uniformly distributed in smaller range compared to the weights them- selves, resulting in smaller errors for INT4 quantization. Furthermore, since intuitively the magnitudes of weight differences are smaller than those of weights themselves (informally sup- posed that ||ōw,]| l|w, w:-lll < [|we]]) and the relative quantization errors are similar be- tween weights and weight differences (informally supposed that [lq(km)) (acdaa))--un Awre [la(w))) the
[
[

---

weight differences compression potentially has a smaller error relative to the weights themselves: [l( (lell))-well where q(-) is the quantization function.
2 ||
||m|||
2) In theory. weight differences compression improves convergence compared to naive weight com- pression. When extended to biased compressors, we present theoretical guarantees for convergence at the same rate ordinary SGD, detailed in Section 4.2. In contrast, demonstrate that using biased compressors directly on weights can lead to convergence failure, as illustrated in an example in Section 4.1, This proves that biased compressors are not compatible with QSDP or ZeRO++.
3.2 Two-Level Gradient Smooth Quantization
3.2.1 Two-level Gradient Quantization (TLq)
As discussed in Section 2.3, the 2-step all-to-all communication strategy benefits gradient communica- tion when quantization applied. However, also introduces error accumulation due consecutive 4-bit bit quantization steps, necessitating additional rounds of training and communication that diminish the per-iteration communication savings. We observe that applying U'Qq, which quantizes gradeints to extremely low precision, such as 4-bit, leads to noticeable deviations in training loss compared to full-precision training, as illustrated Figure 5.
TL4:: Instead of employing a global 4-bit quantization strategy for both inter-node and intra-node all-to-all communications, we propose a two-level precision quantization strategy. This approach balances performance and accuracy by enhancing accuracy without introducing additional overhead. For intra-node all-to-all communication, gradients are quantized INT8 before sending. After receiving the data, each GPU dequantizes the received data back to the full precision (i.e., FP32) for local reduction. This reduced data is then quantized to INT4 to minimize inter-node communication overhead. The detailed methodology is depicted in Figure 3. Since the two all-to-all operations utilize different network bandwidths, their communications can be effectively overlapped (see Table 4).
3.2.2
TLq with Hadamard Smoother (TLq-HS)
While T'4 brings the training loss closer to the baseline, it does not achieve perfect alignment. In gradient quantization, outliers can significantly amplify quantization errors. Although group-wise quantization isolates outliers to minimize their impact on the precision of values in other groups, the values within the same group remain affected.
TLq-HS: To mitigate the outlier issue, we apply Hadamard transform [9] to the gradients before quantization. The Hadamard transform, specific type of generalized Fourier transform, exhibits properties such as H = H and H HT = I, distributing outlier information across nearby elements and effectively smoothing them out. For detailed description of the methodology, see Algorithm 3.
3.3 Performance Optimizations in Implementation
Optimizing Memory Efficiency for Weight Differences Computation: After the all-gather communication, each GPU receives weight differences from the oth- ers, which are then added to the model weights to update them. As discussed in Section 2.1, with ShardedDP enabled in Megatron-LM. each GPU maintains com-
Algorithm 3 TLq with Hadamard Smoother
Require: gradient grad
1: function TLq-HS
2: x Hadamard (graad
3: 49abit +- QuantizeSBit(Q)
4: lisl(ggasst intro
IntraAlltoAll (gggaat)))
5:
list list(gintra Dequantize (list (gAsbit intra))
plete copy of the model weights for forward
6:
(iottraaraa Hadamard (ist (fontra))
and backward computation. This contrasts
7:
Greduced Reduction (ist( ((ittr))))
with ZeRO, where model weights are re- leased after computation. By reusing these locally stored model weights in Megatron- 10:
8:
greduced Hadamard @vedweed
9:
Taabit + Quantizee4Bi( gredwced)
list Sntter InterAlltoA ( (g
LM, our implementation eliminates the
11:
list((ginter) t Dequantize (list (list (g
need for additional buffers to retain model 12:
9
+- Reduction (list (Ginte)))
weights for calculating the differences, thus 13:
Ifénal reduced + Hadamard (g finalll reduced)
enhancing memory efficiency.
5

---

Simplifying Hadamard Transforms: In a naive implementation, the Hadamard transform would be applied at each step before quantization and after dequantization. However, by leveraging the orthogonality of the Hadamard transform, i.e. H H = T, we omit the transform after the intra-node all-to-all dequantization (Algorithm 3, Line 6) and before the inter-node quantization (Algorithm Line 8). Furthermore, by utilizing the distributive property, i.e., Σ; Hg, H H, 9i. we move the second Hadamard transform from after the inter-node dequantization (Algorithm 3, Line 11) to after the final reduction (Algorithm 3, Line 12). These simplifications reduce the unnecessary computational overhead associated with repeated Hadamard transforms.
Fusing GPU Kernels with Group Size Alignment: To mitigate additional data movement from global memory- which typically exhibits the slowest memory bandwidth we fuse the Hadamard transform with the (de))uuntization operations into a single CUDA kernel. This fusion allows the operations to run nearly as fast as the quantization operation alone. It is worth noting that, for this fusion to be efficient, there must be an alignment between the two. Specifically, the size of the quantization group must be divisible by the size of the Hadamard matrix, ensuring that memory traffic remains within the kernel block. We choose H to be small (c.g., 32x32) because, this size, the transform operation on the GPU typically memory-bound and incurs minimal overhead. While larger H sizes offer better smoothing capabilities, we find that a 32x32 matrix is sufficient to effectively smooth outliers in gradients.
4 Theoretical Analysis
4.1 Counterexample of Biased Weight Compression
One of the advantages of our proposed weight difference compression is the compatibility both biased and unbiased compressors. Note that using biased compressors directly on weight compression incurs issues in convergence under standard assumptions, as avoided in QSDP [18] ZeRO++ [32]. We illustrate such issues in the following toy example.
least square problem with 2 minweR* [f(w) = ||=]]]] and stochastic gradient g(w) = (4w1, 0) with probability 0.5 and g(w) = (0, 40,4) T with probability 0.5, thus E[g(w)] = v.f(w)) We use the initial value Winee (1. 1)) the learning rate 17 < 0.125, and the following nearest ternary quantizer: is element-wiss absolute value, and round(.) :) quantizes each element to the nearest value in {-1, 0, 1). It is easy to check that for SGD under such settings, the weights before quantization will be either (1 41/, -1) or (1, (1.-1+ 4n) resulting in g(w=) = (1, T which means that SGD with ternary weight quantization gets stuck at the initial value in this case, while SGD without weight quantization and SGD with weight difference quantization both converge to the optimal.
Counterexample 4.1. Consider
(0,0) 0) T
max( [[w]), (w) round(w/ (w/s) * s, where
4.2 Convergence Analysis
To theoretically analyze the convergence of our distributed training algorithm with communication compression, we focus on the following SGD variant with gradient compression and weight difference compression. We use SGD to solve the following optimization problem: f = min, fww,, where fw) is the objective function, w E Rd is the model parameter.
Algorithm 4 SGD with SDP4Bit
1: Initialize main parameter weights wi
2: Initialize compressed parameter weights wi +- wi
3: for all iteration t [T'] do
4: Compute gradient: 9t-1 V V (1;1-11:0 G-11
5: Compress gradient: -t-1 (g(-1-1)
6: Update main weights: W; + W:-1 1- 7/t
7: Compress weight difference: Å, = Cy Ww wit1)
8: Update compressed weights: V. f- 11 + Å
9: end for

---

Note that we use unbiased compressors for gradient reduction, and arbitrary (potentially biased) compressors for weight collection. We formally define these two classes of compressors as follows.
Definition 4.1 (Unbiased x-approximate compressor [1]). An operator U R + R is a
approximate compressor for 0f [[)( = and [l[((() - =Il S x
x||l|
Ve E Rd.
Definition 4.2 (6-approximate compressor [13]). An operator C R + Rd is 6-approximate compressor for E [0, 1] if Bl(((u) oll2 S (1-5) ()|l
Vu E Rd.
Remark 4.1. Note that, in a certain sense, the class of d-approximate compressors contains the class of unbiased compressors. It is easy to check that any k-approximate unbiased compressor U can be converted - of 6-approximate compressors typically provides more options such as top-k sparsifiers, and top-k low-rank compressors. Thus, we consider arbitrary (biased or unbiased) d-approximate compressors for weight compression in our theoretical analysis.
I+ -pproxxixaa biased compressor C(v) = (x) Furthermore, the class
Remark 4.2. For distributed training with P workers, we define the compressed gradient as 9t (g( 1e E)E(p] Ug( (gt,s, where 9r }o LiE(P) 9t,t, and It, is the stochastic gradient from the ith worker in iteration. We assume that Ug is unbiased и-approximate compressor of the average gradient 9t-
Assumption 4.1. (Smoothness) We assume that ((x) is /-smooth: [V/(x) Vs(i)l T.l
vll. Vx, E Rd which implies f(y) f(x) (Vf(z). x>) + 21l =112.
Assumption 4.2. For any stochastic gradient V (). where Ç is an independent random sam-
ple, assume unbiasedness F fw: wr] V (wr), and bounded variance E (w:;() V [( - pl (
o² ([27), Assumption 3).
We derive the following error bounds on the convergence of SDP4Bit under the above assumptions. All proofs can be found in Appendix A.
Theorem 4.1 (Convergence error bound). For arbitrary non-eonvex functon under Assumption 4. and Assumption 4.2, taking learning rate 1 S 1n Algorithm 4 converges to a critical
point with the the following error bound:
ET (()))(()))) T+1
80T (3 + +p + R) (f(w)) - -5) + V
(11 8)(K + 1) - f()
T 1+1
T+1
Remark 4.3. Note that compared QSDP [18], our convergence analysis does not require Polyak- Eojasiewicz condition or the specific choice of weight quantization (random shift). In other words,
Theorem 4.1 shows that our proposed algorithm has the same O convergence rate as ordinary SGD for general non-convex functions, but under much weaker assumptions compared QSDP.
5
Evaluation
5.1 Experimental Setup
Hardware:
The experiments are conducted on two different clusters to evaluate SDP4Bit across varying network environments: 1) 16 nodes, each node equipped with 4 Nvidia -00-SXXM-40GGB GPUs. All nodes are interconnected with 100 Gbps Slingshot10 network, providing slower inter- node bandwidth. 2) 16 nodes, each node equipped with 8 Nvidia H800-SXMS-80GB GPUs. Each node is connected using 8 InfiniBand links, achieving total bandwidth of 3.2 Tbps, providing higher inter-node bandwidth.
We use 1//lloat2/ (weights/graddients) mixed-precision in Megatron- LLM [26] as our basic Baseline for both accuracy and E2E throughput analysis. Within each set of experiments, we ensure consistent hyper-parameters to ensure faireess. Detailed parameters are provided in Appendix D. Additionally, we implement another baseline for comparison in Megatron-LMM. using the same quantization strategy in ZeRO++, employing 4-bit quantization for both weights (group- wise weight quantization, refered to as qW) and gradients (twice all-to-all with uniform level 4-bit quantization, refer to ULq).
Baselines:
Dataset and Models:
To demonstrate that SDP4Bit does not adversely affect end-to-end training loss, we conduct pre-training GPT-series [23] models ranging from 125M to 6.7B parameters
7

---

Table 1: Final validation loss of pre-training with different quantization strategies.
Weight
Gradient
GPT Model
Baseline
SDP4Bit
qW
qWD
TL4
TLq-HS
125M 350M
2.29392
2.57312
2.29274
2.30479
2.29528
2.29590
2,08719
2.27405
2.08730
2.09551
2.08912
2.08964
1.3B
1.92744
2,04608
1.92881
1.95075
1.93134
1,93238
on the Pile dataset [8], using validation loss as the accuracy measure. Each test runs for 80,000 iterations, processing over 40 billion tokens. For throughput evaluation, select models ranging from 1.3B to 18B parameters, with end-to-end training throughput as the metric. In these tests, the accumulation step is set 1. Note that model parallel is required for models larger than 6.7B, and different tensor parallel sizes are used on A A100 and H800 clusters for models larger than 13B. Please refer Appendix B Table 8 for detailed model parallel configuration.
5.2 Accuracy Evaluation
First, we analyze the impact of SDP4Bit on the accuracy of E2E training- As shown in Table the training results for three different model sizes indicate that the final loss of SDP4Bit is comparable to the baseline, with a maximum increase of only 0.24%. Additionally, Figure details the training curve of GPT-6.7B, demonstrating that the training curve of SDP4Bit perfectly aligns with the baseline. This indicates that the impact of SDP4Bit on accuracy negligible. In contrast, the 4-bit quantization strategy in ZeRO++ (which directly applies quantization weights and uniformly uses 4-bit quantization with all-to-all for gradients) results in significant accuracy degradation.
Next, we break down and analyze each strategy within SDP4Bit. 1) For weight communication reduction, as shown in Table 1, directly quantizing weights (qW) to bits results in validation loss that increase of up to 12% compared to the baseline. In contrast, our weight difference quantization (gWD) method achieves a validation loss nearly identical to the baseline. Notably, we use a consistent quantization group size of 2048 for both tests. 2) For gradient communication reduction, as shown Figure 5, applying Uniform Level quantization (ULq), similar to the method used in ZeRO++. results in significant gap in the loss compared to the baseline. In comparison, our Two Level quantization (TLq) siginificantly mitigate the loss gap between with baseline. Additionally, Figure illustrates the effectiveness of the Hadamard transformation in smoothing outliers. Table 1 and Figure 5 further demonstrate the contribution of the Hadamard smoother to accuracy. Notably, compared TL4, T'Lq-HS further narrows the validation loss gap, making it almost identical the baseline.
2.7
Hadamard Transform
101
Log Trequane
2.6
2.30
Valldation
2.5
72000
76000
80000
3..00
4.00-C4
4.0-05
Roo
Transform
2.4
Baseline
Log Trequency
ULq
TLq
TL
20000
40000
60000
80000
Iteration
-.0040
5.5w-23
D.0
5,2000
LS625
Figure Validation loss comparison for the Baseline, UL4, TL4, and TLq-HS on the GPT-122 model. Uniformly applying 4-bit gradient quantization twice results in noticeable gap compared to the baseline. In contrast, two-level quantization (8-bit for intra- node 4-bit inter-node) mitigates this gap. The Hadamard smoother further reduces gap, making the loss nearly identical to the bascline.
Figure Comparison of gradi- ent histograms before and after the Hadamard transformation. The trans- formation reduces the impact of out- liers, resulting smoother gradient distribution.

---

Table 2: E2E throughputt on different model sizes with std.
4xA100, 16 nodes (Slingshot 10)
8xH800, 16 nodes (InfiniBand)
Model Size
Baseline TFLOPs
SDP4Bit TFLOPs
Baseline TFLOPs
SDP4Bit TFLOPs
Speedup
Speedup
1.3B
24.1 +0.03
57.6 +0.03
x.39x
69.1 +0.96
106.0 +2.66
1.53
2.7B
24.0 #0.00
58.4 #0.07
2.43x
71.9 #0.56
116.9 =0.98
1.63
6.7B
10.8 +0.00
37.1 #0.00
3,44x
26.2 #0.33
77.9 #2.43
2.97
13B
9.7 10.04
26.0 #0.03
2.68
13.9 #0.17
53.5 1.36
3,85
18B
10.2 +0.00
29.8 +0.04
2.92x
14.5 =0.07
59.2 +1.37
4.08
Table 4: Performance of Different Quantization Strategies on GPT-1.3B over 32 A100 with stan- dard deviation.
Table
Final validation loss of GPT- 125M with different group sizes.
Baseline Val Loss
2.29392
Quantization
Grad. Comm
TELOPs
TLq-HS
Group Size
64 2.29537
128
512
Strategy
Time (ms)
Val Loss
2.29528
2.29670
Baseline
379.3 10.34
24.4 10.00
Group Size Val Loss
1024
2048
TLq-HS
45.9 10.03
43.3 10.05
QWD
2.29338
2.29274
Ulq
45.0 +0.03
43.3 +0.04
Group Size
32
128
2048
SDP4Bit
45.8 +0.03
58.5 +0.07
qW
Val Loss
2.39580
2.44712
2.57312
SDP4Bit
64.6 10.05
55.2 10.07
(HS w/o fused)
5.3
Throughput Evaluation
Next, evaluate the improved E2E throughput, measured in FLOPS per second, of SDP4Bit on both hardware platforms. For all tests, the results are averaged over 10 iterations after 20 warm-up iterations. As shown in Table SDP4Bit achieves an E2E training speedup of up to 4.08x. For models with the same model parallel configuration (e.g.. 1.3B and 2.7B; 13B and 18B), both the E2E throughput and speedup from SDP4Bit increase as the model size grows due larger models having higher computational efficiency but also encountering increased communication overhead.
The throughput of the 1.3B, 2.7B, and 6.7B models across the two platforms indicates that SDP4Bit provides a more significant speedup when network bandwidth is lower. This is because lower bandwidth results in higher communication overhead, which SDP4Bit effectively reduces through efficient quantization techniques.
Baseline
SDP4Bit
Weighh
6.78 Model Slingshot10 1 A100
138 Model Slingshat10 A100
GPT-2 78 on 12 A180 GPUs with Singshatt
TRLOPs GPU
40
33.6
57.3
30
27.
26.
26.0
34
12.9
1144
10.5
10
335
112
20.8
24.2
Commi
241
0
0
32
64
32
64
Bacine
Tup
so Nos4
6.7B Model Infini Band H800
138
Model InfiniBand H800
GPT-2.75
125 HB00 GPUs with infiniBand
100
TTLOPs GPU
83.6
73.9
60
37.5
53.5
50
40
502
25
26.4
26.0
25.2
124
13.5
500
32
48
64
96
128
32
48
64
96
128
barelre
1M6
Number GPUs
Number GPUs
Figure 8: Throughput breakdown of SDP4Bit GPT-2.7B.
Figure 7: Scalability SDP4Bit.
In addition, we demonstrate the scalability of SDP4Bit using GPT models of 6.7B and 13B pa- rameters, with tests conducted on up to 128 GPUs, as shown in Figure 7. Under low bandwidth conditions, SDP4Bit achieves an average speedup of 3.40x for the 6.7B model and 2.49x for the 13B model. In bandwidhd InfiniBand environments, the speedup averages 3.08 for the 6.7B model and 3.73x for the 13B model. The comparatively lower speedup for the 13B model under low bandwidth conditions can be attributed to the introduction of pipeline parallelism, which diminishes the proportion of communication handled by ShardedDP. Overall, SDP4Bit consistently maintains stable speedup performance across various GPU numbers and network environments.

---

5.4 Ablation Study
Components Breakdown. Figure 8 demonstrates the throughput improvement of qWD, TLq-HS, and their combination (SDP4Bit) on two different platforms. qWD alone provides a speedup ranging from 1.1x 1.2x, while TLq-HS alone results in an E2E speedup of 1.4x to 1.8x. The notable benefit from gradient quantization stems from the high communication overhead associated with Float32 gradients in baseline training, which is higher compared to B 6 weights. When they applied together, SDP4Bit achieves more substantial speedup, ranging from 1.6x to 2.4x.
TLq-HS ULq. Table 4 compares gradient quantization between TLq-HS and ULq. The results show that although TLq-HS employs 8-bit quantization for intra-node gradient communication, introduces negligible overhead compared to 4-bit communication. This is due to 1) the high bandwidth of intra-node communication and 2) the fact that most intra-node communication is overlapped with the slower inter-node communication.
Hadamard Kernel Fusion. Table shows that, compared to the SDP4Bit without fusing Hadamard Transform kernel, our optimized SDP4Bit reduces gradient communication overhead by 29%. Ad- ditionally, we provide throughput comparison in Table to further illustrate the impact of the Hadamard transformation. The results confirm that our Hadamard kemell fusion effectively reduces the overhead, making the transformation nearly zero-overhead and even matching the performance of quantization without the Hadamard transformation.
Convergence with Different Group Sizes. Table examines the impact various quantization granularities on the end-to-end validation loss during the pre-training of the GPT-125M model. For TLq-HS, a gradient quantization group size of 128 presents sufficient, with smaller sizes yielding no significant accuracy improvements. For qWD, quantization group size of 2048 achieves training accuracy comparable to the baseline. Table also presents the 4-bit weight quantization (qW) while using small group size. It is evident that even with very small group size (e.g., 32), direct 4-bit quantization leads to a significant gap in accuracy compared to the baseline, making 4-bit quantization for weights suboptimal.
6 Related Work
Apart from ZeRO++ [32] and QSDP [18), which specifically designed for communication compression ShardedDP, most previous studies have focused on traditional DP, primarily utilizing gradient compression. This includes both unbiased compression techniques [1, 33, 38, 5), which employ randomized compressors, and biased compression methods with eттоr compensation [12, 31, 30, 29, 24] that require extra storage for residual errors, making them less suitable for resource- intensive training of LLMs. Other strategies like local optimization or federated learning reduce communication frequency rather than volume [16. 28, 35, 34, 2, 20], but increase memory use, complicating their application in LLM training. In addition, techniques like low- fow-precision training [19, 22] and parameter-effice reduce communication. In a different vein, weight quantization for inference has also been explored [7, 6, 39, 37, 4], employing more resource-intense methods compared to those used in training to fine-tune compression parameters.
fine-tuning [10, 3. 14] minimize the volume of trainable variables to
The Hadamard transform has been applied to machine learning data, as seen in HQ-MM's [36] com- pression of activations and THC's [15] gradient communication within a parameter server framework. Unlike THC, SDP4Bit enhances collective communication operations and GPU optimization.
Conclusion
In this paper, we propose SDP4Bit, communication reduction strategy for Sharded Data Parallelism. SDP4Bit reduces both weight and gradient communication to nearly 4 bits while maintaining model accuracy comparable to the baseline. We implemented SDP4Bit in Megatron-LM and optimized it to reduce quantization overhead. Specifically, our experimental results demonstrate training speedup of up to 4.08 x on 128 GPUs. This paper focuses on LLM pre-training, but we plan extend our work to other models and areas such MoE, computer vision, and fine-tuning in the future.
10

---

References
[1] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communication-Effiiinn SGD via Gradient Quantization and Encoding. In NeurlPS, 2017.
[2] Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations. In NeurIPS, 2019.
[3] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. In NeurlPS, 2024.
[4] Peiyan Dong, Lei Lu, Chao Wu, Cheng Lyu, Geng Yuan, Hao Tang, and Yanzhi Wang. Pack- QVT:: Faster Sub-8-bir Vision Transformers via Full and Packed Quantization the Mobile. In NeurIPS, 2024.
[5] Fartash Faghri, Iman Tabrizian, Ilia Markov, Dan Alistarh, Daniel M Roy, and Ali Ramezani- Kebrya, Adaptive Gradient Quantization for Data-Parallel SGD. In NewrIPS, 2020.
[6] Elias Frantar and Dan Alistarh. SparseGPT: Massive Language Models Can be Accurately Pruned One-Shot. In ICML, 2023.
[7] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: ACCURATE POST- TRAINING QUANTIZATION FOR GENERATIVE PRE- PRE-TRIINED TRANSFORMERS, In ICLR, 2023.
[8] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprini arXiv:2101.00022 2020.
[9] A Hedayat and Walter Dennis Wallis. Hadamard matrices and their applications. The annals of statistics, pages 1184-1238, 1978.
[10] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2021.
[11] Jajjun Huang, Sheng Di, Xiaodong Yu, Yujia Zhai, Jinyang Liu, Yafan Huang, Ken Raffenetti, Hui Zhou, Kai Zhao. Zizhong Chen, et al. gZCCL: Compression-Acceleraaa Collective Communication Framework for GPU Clusters. arXiv preprint arXiv:2308.05/99 2023.
[12] Sai Pranecth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error Feedback Fixes SignSGD and other Gradient Compression Schemes. In ICML, 2019.
[13] Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian U. Stich, and Martin Jaggi. Error Feedback Fixes SignSGD and other Gradient Compression Schemes. In ICML., 2019.
[14] Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-Efficien Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization. In NeurIPS, 2024.
[15) Minghao Li, Ran Ben Basat, Shay Vargaftik, ChonLam Lao, Kevin Xu, Michael Mitzenmacher, and Minlan Yu. THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression. In NSDI, 2024.
[16] Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't Use Large Mini- batches, Use Local SGD. In ICLR, 2020.
[17] Nya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv preprint 1rXiv:1711.051000 2017.
[18] Ilia Markov, Adrian Vladu, Qi Guo, and Dan Alistarh. Quantized Distributed Training of Large Models with Convergence Guarantees. In ICML, 2023.
[19] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed Precision Training, arXiv preprint 17X1v:1710003740, 2017.

---

[20] Giorgi Nadiradze, Amirmojtaba Sabour, Peter Davies, Shigang Li, and Dan Alistarh. Asyn- chronous Decentralized SGD with Quantized and Local Updates. In NeurlPS, 2021.
[21] Pitch Patarasuk and Xin Yuan. Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations. Journal of Parallel and Distributed Computing, 69(2): 17-124, 2009.
[22] Houwen Peng. Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang. Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jía Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, and Peng Cheng. FP8-LM: Training FP8 Large Language Models. arXiv preprint arXiv:2310.18313 2023.
[23] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
[24] Peter Richtárik, Igor Sokolov, and Ilyas Fatkhullin. EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback. In NeurlPS, 2021.
[25] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT. In AAAI, 2020.
[26] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.0800 080533, 2019.
[27] Sebastian U. Stich and Sai Praneeth Karimireddy, The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication. JMLR, 2020.
[28] Sebastian Urban Stich. Local SGD Converges Fast and Communicates Little. ICLR, 2019.
[29] Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit Adam: Communication Efficient Large-SScale Training with Adam's 's Convergence Speed. In ICML, 2021.
[30] Hanlin Tang, Chen Yu, Xiangru Lian, Tong Zhang, and Ji Liu. DOUBLESQUEIZE: Parallel Stochastic Gradient Descent with Double-pass Error-Compensated Compression. In ICML, 2019.
[31] Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization. In NeurIPS, 2019.
[32] Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, and Yuxiong He. ZeRO++: Extremely Efficient Collective Communication for Giant Model Training. In ICML, 2023.
[33] Hongyi Wang, Scott Sievert, Shengehao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright. ATOMO: Communication-efficieee Learning via Atomic Sparsification. In NeurlPS, 2018.
[34] Blake Woodworth, Kumar Kshitij Patel, and Nathan Srebro. Minibatch vs Local SGD for Heterogeneous Distributed Learning. In NeurIPS, 2020.
[35] Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai. Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local SGD better than minibatch SGD? In ICML, 2020.
[36] Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training Transformers with 4-bit Integers. In NeurIPS, 2023.
[37] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wa, Julien Demouth, and Song Han. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In ICML, 2023.
12

---

[38] Jihao Xin, Marco Canini, Peter Richtárik, and Samuel Horväth. Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees, arXiv preprint arXiv:2305. 18627, 2023.
[39] Zhewei Yao. Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yux- iong He. ZeroQuant: Efficient and Affordable Post- Bost-Training Quantization for Large-Scale Transformers. In NeurIPS, 2022.
13

---

Appendix
A Proofs
We the following lemma (simplified from [27], Lemma 14) without proof.
Lemma A.I. For every non-negative sequence {ra)22 and any parameters there exists constant S such that
T
- T++1 + 0n)
1 To T+1
TT+1
aTo
2/era
en S T VT+1
Theorem 4.1 (Convergence error bound). For arbitrary non-convex fuuntinnn under Assumption 4.1 and Assumption 4.2, taking learning rate 7 S 13p(ttpttptx] Algorithm 4 converges critical point with the following error bound:
Y ((())( T+1
S0L (3 + pr + p n) (f(wo) - 8*)
(11 8)(x - (f - f*)
+ 10
T+1
T+1
Proof. By using smoothness (Assumption 4.1), we have
f(w(+1) & f(w)) n (V)()) T
(1,(1)111
Taking expectation w.r.t. the random compressor Uy, we have
LL
< S((w)) 17 (V](w)),9t) +
Fe 11 (gc))l
112
=[(w)) - - (V](w)),g) gr) +
[|lga|12 Bog (gx) ge 2
2
17 L(x + 1)
S fw(()) - (Vf(w)) +
11
2
14

---

Conditional on wt, taking expectation on the random sample Se, we have
{((w)) ( TfT[(N= 1)
Eellgel1
1fT((1 1)
f(()) 1 (V](w)), Vf(())
By Bell9t V[((a))+[[())) V
1f 1)
= f(w)) (V)(w)), (.)())))
Bclll9: V)(ūi)) |2 [V)
2
Lf 1)(p + 1)
fV 11 L + 1)o²
f(w) 1. (w;), (.,))
2
S f(w) L. (w;)) [.,) L () + [Vf(ū;)]]
17 L + 1)o²
= f(w)) 14 (Vf(w)) f(.))) + V v.((i))) + 22 1) f()))
2
L + 1)o²
2
((..)) - - L + 1)(0 + |V [
-((w)) 1)
V[))), V/))
2
1f2 + 1)0²
2
= =((w)) - (1
(1-TL(n
nL(n + 1)(
- -V())) Vf(ū.))
2
1/ + 1)o²
2
S {((w)) - "1-
"7T((x + 1)(p+
[ " (Ivs(())- -
2
? +1 IVV((11)) L + 1)0²
D (a, s 2/10112 + 211611
2
+ (x² 1)0²
" [1 nL(K + 1)(p 1)))) " (2 f(w)
2
Again using smoothness, and taking 1) S M[(++[[++1] we have - 1/1(( + 1)(p + 1)) S - and, we have
Be Bge [ (wx+1))] 1)]
17 nL(L + (o0p=++)()+11)) (V
nL
[w u 12 L + 1)o²
$ f(w)
2
2
1/ [1 - (L( + 1)(p + 1)] fw, + 11
||e||2 + 2 + 1)o²
$ f((,)
2
2
1 V ((x)) + ?Lº
1/2 + 1)o²
< [w.)
2
2
where we define the sequence
e, = Wi We, eo
Now we establish the upper bound of the sequence || as follows.
First, using W1+1 = wi -n]d and w w; + C.n (W++1 ...) we have the following equations:
W++1 Wi+1 6t-1 We w. (9)) C. (wi+1 wi) E, n1gg Ce. n91)(
15

---

Taking expectation w.r.t. the random compressor Cw. we have
E Ewelllet+11 [1
Ruel - (pbg)(0)) Cer - (Mg (x))))
(1 8 1)(g
Taking expectation w.r.t the random compressor Ug. we have
E ge B [eeeelee |lllll [|]]]
s (1 8) 8) [lle - npH4(19))1
(1 5) B) - 119t + 199: r (gr, ) )12]
= (1 8)llec - n1::1l² + - - I1 (g))))1
$ (1 5)ee - =?9 + (1 x)n)"x1l9el
Conditional on wt: taking expectation on the random sample Ge. we have
Eg [Eeclllll
Ewelll et> |
s (1 +
-
= (1 d)lle: - (1 - -100---0000111110000000000000110
< (1
(1 5)||e
1)1/20²
With vb > 0, we have
Eg [B
$ (1 - (1 + b) + +1 - + ">> + - p +0)
+ (1-50)x + 1)7700
= -1 - () b)]le (1 6) + b- + (px + p 15) [
5) + 13
Then, by taking b 2(1-5) we have (1 - 5-55)1111 5) = 1- / 1 2- < 3/ and
Eç [Ec Ewe ||lee111121 111
s (1 " Ileal1 + +(1 (1-88)
(1- 6)n? (3 + ph + k²) [[ff(()))) + +1 (1- 8) + 1)n²
We simplify the notation by denoting B[[le++1|| 1 = Ee Eg Ewe |11111111/ [1]]]] and then unroll the sequence of e, back to = O.
Ellle++11 [13]
(2
" y0- (1 (1-5)y) 5yy2
+ px + ( (k- 1)7
[[Vf(0+)]1
2
1
8
8 /-r
S1 8)n?
+ pk p+
Σ
| (1- 1))720 Σ
8
2
2
T=0
8 2
2(1 8) 1)m?20
S1 - 6)n?
pk + app K
Σ
8
==0
E - -6g- <
1
1-(1-
16

---

Taking 17 $
we have
10t(3 13t(3tac))
B|1ee+1118]
B
-
2(1 -8)(n + 1)170
(1
| (V
10002 (3
pr
+
p
2
810L (3 ps + p x)
7=0
1-8 8
- 2(1-5)(H
+ 1)11²²
t-
100L22
2
810L
7=0
(1 (1- 8)5
8
(1-5)( + 1)mo?
Σ
2007.
2
10L
T=U
Then, stacking B[[le]]] || and taking total expectation, we have
Blll Blle++11 e++ [19]
(1 -26
+ (T+ ((
8
110-1)(())))))) -0)(x + 1)mo²
Σ
2
2001
10L
T=0
(1-
T
+000
8
(T + 1)(1 - 8) + 1)no?
[vf((t)) +
200L
10L
1=0
1==
1-8 8
T
(T + 1)(1 5) + 1)n0²
[V +
100T.
10L
1=0
Putting all the ingredients together and taking total expectation, we have
T
B[f(wr+1)
0=0
T
T
nL2 T +
(T + 1)n + l)o²
" B[f(w))] -
Σ
E
Σ Bl|le/191
4
2
2
t=
t=0
1=0
[[((w-I)]
T Σ
2L T
(T 1 L(s + 1)0² 2
!1 < <B([(w))]
+ B [(
[llley [13]
4
2
1=0
t=
E[f(wr-]]
T
+ nT?
T
(T + 1 [(K + 1)o²
<B[[[wo)]]
1 B[|B)((0))
[1]
4
2 t=
2
E[f(WT-1]]
$ <B[[(wo)]
11
Σ
1
Blll [[ (1 8)m
Σ
[(x))))
200
1=
1=
(T (T(1)(1 - 6)( + (1+1)(118((6)
1)
(T + 1)m² L(x + 1)o²
20
([s(wr- 1)] < [[](w)))
"
Σ
T
Blll [[] || +
(T + 1)(11
8) + L)
8
20
0=
T
B[f(
[[f(WT-
(11 5) + L
Σ E[
8 +1)
T+1
77
20
0=
17

---

Finally, using Lemma A. we have
(
T+1 t=
B[f(wo)] + + E[f(wr+1)] 8(11 (n + 1)T
T+1
20
80L ( + pa + k) (f(wo) - f*) + 4
(11 0) [)L(
++1
T+1
B Other Evaluation Results
To further demonstrate the effectiveness of SDP4Bit in enhancing training efficiency, we present the relationship between wall clock time and training loss in Figure
2.5
Baseline
SDP4Bit
2.4
ZeRO++
Validation Loss
2.1
2.0
1.9
1.8
20
40
60
Wall Clock Time (hours)
Figure 9: Comparison of validation loss wall-clock time for Baseline, ZeRO++ and SDP4Bit the GPT-6.7B model.
To further illustrate the impact of the Hadamard transformation on (de)quantization performance, we provide (de)quantization throughput experiment in Table which is tested on an A100 GPU.
Quantization
Dequantization
Input/Output Size
w/o Had.
w/ Had.
w/o Had. w/ Had.
8 MB
305.6+10.99
30..8+10.6
367.7=10.6
359.6::9.66
16 MB
389.0 12.8
387.1 8.2
42..0=10.6
428.6:77.6
64 MB
494.8+ 3.7
493.7
2.6
505.3 2.1
505.6:22.2
512 MB
682.1 0.8
681.6#
1.2
685.1=
0.8
685.2400.6
1024 MB
686.5 1.2
686.31:
0.4
688.0= 0.3
688.0+0.3
2048 MB
688.6 0.2
688.61
0.2
689.5= 0.2
689.4400.2
Table 5: (De)quantization Throughput with/without Hadamard, including std. dev.
C
Notations in Training
gW
original ZeRO++ int4 weight quantization
gWD
weight difference int4 quantization
UL4
original ZeRO++ uniform-lewe level Int4-Int4 all-to-all gradient quantization
TLq
two-level Int8- Int4 all-to-all gradient quantization
TL4-HS
two-level Int8-Int4 all-to-all gradient quantization with Hadamard Smoother
Table 6: Notations in experiments.
18

---

D Detailed Training Settings
In the experimental section, we utilize a total of six different sizes of GPT models. Their model configurations are detailed in Table 7.
For the accuracy experiments, standardize the batch size to 256, and set sequence length to 2048. We use AdamW [17] optimizer in all the experiments. The detailed training parameters are listed in Table 9.
In the throughput experiments, to more clearly study the communication bottleneck and ensure consistency across different GPU counts, set the accumulation step to 1. The batch size is adjusted according to the number GPUs, and the sequence length (micro batch) is uniformly set 2048. Due to the different number of GPUs per node in the two architectures, we adjusted the tensor parallel size (TP) and pipeline parallel size (PP) accordingly, referencing [26], to achieve the highest throughput. Specifically, the maximum tensor parallel size is 4 for the 4xA100 environment and 8 for the 8xH800 environment. See detailed parameters in Table 8.
Table 8: Parallel Configuration for Through-
Table 7: Model Size Parameters
put Test
Model Size
Sequence Length
Hidden Size
Layers
Accumulation
125M
2048
768
12
Model Size
TP
PP
Step
350M
2048
1024
24
1.3B
I.3B
2048
2048
24
2.7B
6.7B
2048
4096
32
6.7B
4
13B
2048
3120
40
13B
4/8
2/1
18B
2048
6144
40
18B
4/8
2/1
Table 9: E2E Convergence Training Parameters
Model Size
Learning Rate
Betas
Epsilon
Weight Decay
Batch Size
125M
0.9,
350M
364
0.9, 0.95
Ie-8
0.1
256
1.3B
2e-4
0.9. 0.95
le-8
0.1
256
6.7B
12e-5
0.9, 0.95 0.95
le-8
0.1
256
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction accurately shows the contribution in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In paper title, Section Introduction, and Section Conclusion, we state that this work focuses on the specific problem of communication compression of ShardDP on the training of LLMs. Furthermore, in Section 2.2 Quantization, we state that this work only considers symmetric linear quantization, and in Section 6 Related Work we explain that there are other types of compression methods that are not used in our algorithms. In Section 5.1 Experimental Setup, we list the models and the dataset we used in the experiments.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
20

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The full set of assumptions could be found in Section 4.2 Convergence Analysis. A complete proof could be found in Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The detailed experiment configurations and hyperparameters could found in Section 5.1 Experimental Setup, and Appendix B, respectively. Furthermore, the source code could be found in the supplementary materials, with README file containing the instructions to run our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
21

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The dataset could be downloaded from hugginface website. The source code could be found in the supplementary materials, with README file containing the instructions to run our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips ca/
public/ rublic/guides guuides/CodeSuuiiee stt.iiisssiiiccio for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.co wwtpccccccpcccccccccccccccccccct guides/ /CodeSubmii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All of these detailed hyperparameters and configurations could be found in Section 5.1 Experimental Setup, and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the experiment results with standard deviations in Section 5 Eval- uation. For the figures and plots, we found that the standard deviations and error bars are barely visible since they are too small compared to the average values.
22

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The detailed information about the computer resources could found in Section 5.1 Experimental Setup, Hardware.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://neet //tpp/r///////p///// cwtppulice/tticsssiiiiiiieeeeeie weleeGiiieeiing dnes??
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics, and we do not think this work would have any negative social impact potential harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
23

---

Answer: [NA]
Justification: The goal of this work is to make model training faster with the model accuracy aligned with the original one. This work does not publish any new dataset model or the corresponding new application. This work simply proposes new algorithms that accelerate the training of existing models, thus having no social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release any new dataset or new model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package or dataset, and included the license in the source code package.
Guidelines:
24

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included the license in the source code package, which is aligned with the code packages that our work is based on.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
25

---

Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "1XtRWvh2VT", "reviewer": "Reviewer_Ah9Z", "summary": "The paper introduces a novel approach to reduce communication overhead in Sharded Data Parallelism (ShardedDP) for training large language models (LLMs). It proposes two key techniques: quantization on weight differences and two-level gradient smooth quantization, which effectively compress weights and gradients to nearly 4 bits without compromising training accuracy. The method is implemented within the Megatron-LM framework and achieves up to 4.08× speedup in end-to-end throughput on 128 GPUs. The paper also provides theoretical guarantees of convergence and demonstrates negligible impact on training loss for models with up to 6.7 billion parameters.", "questions": "1. Why is 4-bit quantization needed? Is 8-bit quantization not sufficient in practice, especially since we don't expect foundation models to be trained too frequently?\n\n2. Why is inter node communication quantized to 4 bits while intra node communication is quantized to 8 bits and not the other way around?\n\n3. I think the Hadamard transform matrix might need to be post multiplied by a diagonal matrix with randomly chosen +1/-1 values on its diagonal for it to be effective at removing outliers and flattening the data (https://arxiv.org/abs/1011.1595). Is that what has been done here? If not, why?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Weight difference and two-level gradient quantization are more effective at retaining accuracy and significantly improving throughput than baseline approaches\n\n2. The authors present a complete story with a new quantization algorithm, accompanying GPU kernel implementations, and detailed evaluation/ablation over a range of models and baselines which together clearly illustrate all the benefits of each component of their approach.", "weaknesses": "1. The connection between the convergence analysis and the quantization schemes is not entirely clear. Specifically, it is not clear if the quantizers correspond to the biased/unbiased compressors in Algorithm 4 and if so which of them is biased and which is unbiased. It appears from lines 5 and 7 of algorithm 4 that TLq is unbiased and qWD is biased but I couldn't find any mention/proof of that, so I am not sure if that is the case or if I am missing the point entirely.\n\n2. Most of the empirical analysis is limited to the older generation of GPT models, and it is possible that may not extend as well to modern models like Mistral series, Phi series, Orca series, Llama series etc as well as the newer generation of GPT models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n**W1: The connection between the convergence analysis and the quantization schemes is not entirely clear. Specifically, it is not clear if the quantizers correspond to the biased/unbiased compressors in Algorithm 4 and if so which of them is biased and which is unbiased. It appears from lines 5 and 7 of Algorithm 4 that TLq is unbiased and qWD is biased but I couldn't find any mention/proof of that, so I am not sure if that is the case or if I am missing the point entirely.**\n\n**A:** The definition (notation) of the compressors $\\mathcal{U}$ and $\\mathcal{C}$ can be found in Definitions 4.1 and 4.2 on Page 6, right below Algorithm 4. The compressor $\\mathcal{U}$ in Algorithm 4 Line 5 and Definition 4.1 is unbiased, assuming $\\mathbb{E}[\\mathcal{U}(v)] = v$. The compressor $\\mathcal{C}$ in Algorithm 4 Line 7 and Definition 4.2 is (potentially) biased, as it does not assume unbiasedness. In the proof, the usage of Definition 4.1 can be found in Line 476 and Line 484 of the Appendix A, while the usage of Definition 4.2 is found in Line 483 of the Appendix A.\n\n**W2: Most of the empirical analysis is limited to the older generation of GPT models, and it is possible that may not extend as well to modern models like Mistral series, Phi series, Orca series, Llama series, etc., as well as the newer generation of GPT models.**\n\n**A:** The reason that we chose to use the open-source GPT-2 is that the main purpose of this paper is to accelerate the pretraining of LLMs, which is more time-consuming and thus benefits more from communication compression compared to fine-tuning tasks. The hyperparameters of a GPT-2 baseline pretrained on the Pile dataset are readily available in open-sourced repositories such as GPT-Neo. However, for models like Mistral and Llama, they are typically used for fine-tuning rather than pretraining from scratch on public datasets. The corresponding recipes (hyperparameters) and datasets for pretraining these models are not publicly available, making it challenging to establish a fair baseline.\n\n### Questions\n\n**Q1: Why is 4-bit quantization needed? Is 8-bit quantization not sufficient in practice, especially since we don't expect foundation models to be trained too frequently?**\n\n**A:** The necessity for 4-bit quantization arises primarily from the need to reduce communication overhead, especially in scenarios with limited bandwidth. The choice of quantization level is influenced by the physical hardware and network infrastructure available:\n\n- **Weak Hardware:** For example, some NVIDIA GPUs, particularly the more affordable variants, lack high-speed NVLink and instead rely on PCIe for intra-node communication. This setup can significantly benefit from higher compression rates to mitigate slower communication channels.\n- **Heavy Communication Overhead:** In our communication overhead breakdown experiment shown in Figure 7, we observed that communication time (1142ms) constituted 76.7% of the total overhead compared to computation time (347ms) when training the GPT-2.7B model on 32 A100 GPUs equipped with 100Gbps slingshot10 inter-node bandwidth. Even with 8-bit quantization, the communication overhead remains substantial, as detailed in the communication overhead results **provided in the table below**. Hence, further compression to 4-bit levels can provide additional reductions in overhead.\n\nAdditionally, while full retraining of foundation models may not be frequent, there are scenarios of continual training where incremental improvements or adaptations to new data are necessary. In these cases, efficient communication remains critical, making 4-bit quantization a practical solution.\n\n| Quantization Scheme (Weight/Gradient) | Computation | Communication | Communication Overhead |\n|---------------------------------------|-------------|---------------|------------------------|\n| Baseline (BF16/FP32)                  | 347 ms      | 1142 ms       | 76.7%                  |\n| Quantize to 8-bit (Int8/Int8)         | 347 ms      | 420 ms        | 54.8%                  |\n| SDP4Bit (Int4/Int4)                   | 347 ms      | 241 ms        | 41.0%                  |\n\n**Q2: Why is inter-node communication quantized to 4 bits while intra-node communication is quantized to 8 bits and not the other way around?**\n\n**A:** Intra-node communication typically uses high-speed communication media such as NVLink or at least PCIe, which is typically much faster than inter-node networking. Since inter-node is slower, it naturally becomes the major bottleneck and thus requires a higher compression ratio.\n\n**Q3: I think the Hadamard transform matrix might need to be post-multiplied by a diagonal matrix with randomly chosen +1/-1 values on its diagonal for it to be effective at removing outliers and flattening the data ([source](https://arxiv.org/abs/1011.1595)). Is that what has been done here? If not, why?**\n\n**A:** We didn't multiply the Hadamard matrix by a random +1/-1 diagonal matrix in this paper. The flattening power mainly comes from the original Hadamard matrix itself, which mixes the neighboring elements in the input matrix. The diagonal matrix only adds some randomness to the transformation, which may potentially enhance the flattening ability slightly. However, it also results in additional communication overhead because the receiver of the compressed message also needs the same +1/-1 diagonal matrix as the sender for dequantization. Furthermore, the additional random diagonal matrix incurs some difficulty to the optimization of the fused cuda kernel of Hadamard transform and quantization/dequantization. Thus, we chose to use the simplest form of Hadamard matrix without randomness.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vJ4x2MUvMW", "reviewer": "Reviewer_MiBn", "summary": "This paper proposed a 4bit quantization framework for sharded data parallelism training. It proposed to quantize the weight differences between iterations as the first method to reduce the accuracy degradation. It also proposed to mixed-precision quantization for intra- and inter- node communication. Experiment results show that the proposed system can achieve 2x to 4x speedup on training GPT models with size from millions to a few billions.", "questions": "Please refer to the session \"Weakness\"", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of communication bottleneck in FSDP, especially multi-node training is important. The paper aims to solve this problem and  the results seems to encourage this approach.\n\n2. The paper implemented the framework in real hardware environment and achieves 2-4x speedup by reducing the communication.", "weaknesses": "1. The experiment results shows training/validation loss on Pile for the proposed method and baseline (FP) and QSDP and other method. The loss is an auxiliary variable to the performance of the model. A direct comparison on real dataset will be more convincing. For example, what is the accuracy of the model training by proposed methods and ZeRO/QSDP on mmlu and hellaswag/piqa/winogrande/ARC/ARE etc reasoning tasks, and gsk8k etc math tasks? This is critical to show the accuracy/precision improvement over other 4-bit communication methods.\n\n2. The paper shows the speedup of the proposed methods, but I  would like to know which results in Table 4 is the training throughput for ZeRO++ and QSDP (if applicable)? I would like to see a comparison for accuracy-training_speed between the proposed method and ZeRO /QSDP. Furthermore, is there any results on the training throughput (tokens/sec) at certain sequence length (instead of training TFLOPs)?\n\n3. The paper introduces a few method, including qWD, two-level quantization, Hadamard transforms, etc. I would like to know which ones are the invention of the paper? For example, using Hadamard transform to handle the outlier has been known for a while.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n**W1: The experiment results show training/validation loss on Pile for the proposed method and baseline (FP), QSDP, and other methods. The loss is an auxiliary variable to the performance of the model. A direct comparison on real datasets would be more convincing. For example, what is the accuracy of the model trained by the proposed methods and ZeRO/QSDP on MMLU and Hellaswag/PIQA/Winogrande/ARC/ARE, etc., reasoning tasks, and GSK8K, etc., math tasks? This is critical to show the accuracy/precision improvement over other 4-bit communication methods.**\n\n**A:** We thank the reviewer for suggesting these evaluation methods for a more comprehensive comparison. We use validation loss as a measurement to align with our baseline papers ZeRO++/QSDP. Furthermore, we believe that the loss, as first-hand information, is one of the best metrics to reflect the influence of communication compression on the training procedure.\n\n**W2: The paper shows the speedup of the proposed methods, but I would like to know which results in Table 4 represent the training throughput for ZeRO++ and QSDP (if applicable)? ...**\n\n**A:** \n- **ZeRO++ Results in Table 4:** We thank the reviewer for highlighting this point. We acknowledge that the use of numerous abbreviations might have caused some confusion. The term \"Int4-Int4\" in Table 4, as defined on lines 277 and 292, represents the configuration used in ZeRO++ when gradient quantization is enabled. Specifically, \"Int4-Int4\" denotes 4-bit gradient compression for both intra-node and inter-node communication, which aligns with ZeRO++'s quantization strategy.\n\n- We did not include the throughput results for ZeRO++ with both weight and gradient quantization enabled because the resulting accuracy degradation was substantial. As shown in Figure 1 and line 291-292, the configuration with W4 (4-bit weight quantization) combined with Int4-Int4 (4-bit 2-level gradient quantization) exhibits a significant accuracy gap compared to the baseline, making it impractical for pre-training tasks where accuracy is critical.\n\n- **Accuracy-Training Speed Comparison:** The comparison of accuracy versus wall-clock time between our proposed SDP4Bit method and ZeRO++ (W4&Int4-Int4) for GPT-6.7B training on 128 H800 GPUs is presented in **Figure 1** of the rebuttal document.\n\n- **Training Throughput in Tokens/Sec:** While we primarily report training throughput in TFLOP/s, these results can also be presented in tokens/sec. The detailed corresponding results are provided below:\n\n| Model Size | 4xA100 Baseline (tokens/sec) | 4xA100 SDP4Bit (tokens/sec) | 4xA100 Speedup | 8xH800 Baseline (tokens/sec) | 8xH800 SDP4Bit (tokens/sec) | 8xH800 Speedup |\n|------------|---------|----------|---|---------|-----------|-----|\n| 1.3B       | 169,723.8                | 406,228.9               | 2.39$\\times$           | 974,432.5                 | 1,498,441.4              | 1.54$\\times$           |\n| 2.7B       | 85,886.9                  | 209,101.9               | 2.43$\\times$           | 514,489.8                 | 836,600.1                | 1.63$\\times$           |\n| 6.7B       | 63,950.4                  | 220,223.3                | 3.44$\\times$           | 310,965.6                 | 923,946.0                  | 2.97$\\times$           |\n| 13B        | 60,795.3                  | 161,877.3               | 2.66$\\times$           | 172,901.7                 | 666,543.1                | 3.86$\\times$           |\n| 18B        | 44,843.3                  | 130,724.9               | 2.92$\\times$           | 126,892.3                 | 519,622.9                | 4.09$\\times$           |\n\n**W3: The paper introduces a few methods, including qWD, two-level quantization, Hadamard transforms, etc. I would like to know which ones are the invention of the paper? ...**\n\n**A:**\nOur proposed SDP4Bit is a system-algorithm co-design where the qWD (quantization on Weight Difference) mainly contributes to the algorithm part, while the TLq-HS (Two-level gradient quantization and Hadamard Smoother) contributes to the system part.\n\n**Our primary innovation** is the introduction of the **qWD method**, which involves quantizing weight differences instead of the weights themselves. To the best of our knowledge, this approach is novel and has not been proposed in prior works. We have provided theoretical proof of its convergence. The qWD method significantly reduces the weight communication overhead from the state-of-the-art 8-bit level (used in QSDP and ZeRO++) to a 4-bit level without compromising accuracy.\n\n**For TLq**, we introduced a mixed-precision approach (Int8-Int4) to achieve an optimal trade-off between accuracy and speed. While ZeRO++ utilizes Int4-Int4 quantization, our method combines 8-bit and 4-bit quantization to enhance accuracy.\n\n**Hadamard transform:** We acknowledge that the Hadamard Transform is a known technique for handling outliers and does not constitute a novel contribution in itself. However, our work makes significant contributions in minimizing its overhead. We achieved this by:\n  1. Eliminating the need for two consecutive Hadamard Transforms, as discussed on page 5, lines 183-186.\n  2. Integrating a Hadamard Transform with nearly-zero overhead within the quantization kernel. This optimization is detailed in Section 3.3, lines 190-210, and is evidenced by the similar overheads reported for the Int4-Int4 and TLq-HS configurations in Table 4. We also include detailed throughput results tesed on A100 GPU in our **rebuttal PDF table 2**.\n\n**Comprehensive compression strategy for shardedDP:** Combining all three methods, we established a unique system-algorithm co-design communication pattern for the sharded data parallelism (shardedDP). Our design is crucial for the efficient and scalable training of large-scale language models (LLMs) and is the first to reduce communication overhead of both parameter weight and gradient in shardedDP to nearly the 4-bit level without compromising accuracy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Np1Rh7KtpO", "reviewer": "Reviewer_6CdP", "summary": "Thank you for submitting your paper to Neurips 2024. The paper proposes SDP4Bit, a communication quantization method that mitigates accuracy loss by weight difference quantization and 8-bit intra-node and 4-bit internode quantization. The authors provide convergence analysis to show that SDP4Bit is compatible with both biased and unbiased compressors and experimental results to show that SDP4Bit outperforms existing solutions.", "questions": "1. Under what assumptions are weight differences generally easier to quantify? I've read several papers based on this intuition, but none that provide a comprehensive analysis of this intuition.\n2. If you only need to assume that the magnitude of the weight difference is smaller than the weight, how do you ensure this?\n3. If you ensure this via the learning rate, I think it will potentially lead to higher wall clock convergence time.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors provide convergence analysis to show that weight difference quantization is compatible with both biased and unbiased compressors.\n2. The two-level quantization method is suitable for the existing GPU cluster network topology.\n3. The authors' implementation uses several system optimizations.", "weaknesses": "1. Missing key ablation studies. The two proposed strategies in this paper are weight difference quantization and two-level int8-int4 all-to-all gradient averaging. So I suggest the authors add the following two experiments to compare the accuracies.\n- Running ZeRO++ with two-level int8-int4 all-to-all gradient averaging and its original quantization method\n- Running ZeRO++ with its original int4-int4 all-to-all gradient averaging and the weight difference quantization.\n2. The figures are too small to read.\n3. Related work could add AG-SGD, which compresses changes in activations rather than the activations themselves, so it does not rely on the assumption that gradients are unbiased.\n[1] Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n**W1: Missing key ablation studies. The two proposed strategies in this paper are weight difference quantization and two-level int8-int4 all-to-all gradient averaging. So I suggest the authors add the following two experiments to compare the accuracies.**\n- *Running ZeRO++ with two-level int8-int4 all-to-all gradient averaging and its original quantization method.*\n- *Running ZeRO++ with its original int4-int4 all-to-all gradient averaging and the weight difference quantization.*\n\n\n**A:** \nWe do have ablation studies for weight-only quantization and gradient-only quantization in Table 1 and Figure 5 (and we apologize for any potential confusion caused by the abbreviations). By separating the evaluation of gradient and weight quantization strategies, we avoid the influence of quantization on other parts of communication, and provide a clear understanding of the effectiveness of our proposed method.\n\nInstead of the two suggested experiments, we have the following ones:\n- *Running with full-precision weight communication: compare the proposed two-level int8-int4-Hadamard all-to-all gradient averaging (denoted as TLq-HS) with the original ZeRO++ two-level int4-int4 all-to-all gradient averaging (denoted as Int4-Int4)* (**Figure 5 after Line 313**)\n- *Running with full-precision gradient communication: compare the proposed int4 weight difference quantization (denoted as qWD) with the original ZeRO++ int4 weight quantization (denoated as quant-W4)* (**Table 1 at the top of Page 8**)\n\nWe also acknowledge that there was no direct comparison between our proposed gradient quantization and the original ZeRO++ gradient quantization without the influence of Hadamard transform. To address this, we include the Int8-Int4 all-to-all gradient averaging (denoted as TLq) results in **Figure 2 of rebuttal PDF**, which will also be included in the revised manuscript.\n\n**W2: The figures are too small to read.**\n\n**A:** \nWe acknowledge the issue with the figures being too small to read. We will increase the size of the figures in the revised version to enhance readability.\n\n**W3: Related work could add AG-SGD, which compresses changes in activations rather than the activations themselves, so it does not rely on the assumption that gradients are unbiased. [1] Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees**\n\n**A:** \nThank you for pointing out the relevant work on AG-SGD, which compresses changes in activations rather than the activations themselves. We will include this work in our related work section.\n\n### Questions\n\n**Q1: Under what assumptions are weight differences generally easier to quantize? I've read several papers based on this intuition, but none that provide a comprehensive analysis of this intuition.**\n\n**A:** \nIn our paper, \"easier to quantize\" refers to achieving lower relative reconstruction error during quantization. This intuition is based on the observation on the real data, as illustrated in Figure 4 Page 4. The data distribution in Figure 4 shows that: 1) the quantization on weight differences achieves a finer granularity (smaller gaps between quantization levels) compared to quantization on weights themselves; 2) the weight differences are smaller than weights in magnitude or range, thus combining with the informal analysis in Line 136-143 of Page 4, we intuitively argue that weight differences potentially has lower reconstruction error relative to the weights themselves, which motivates our algorithm design.\n\n**Q2: If you only need to assume that the magnitude of the weight difference is smaller than the weight, how do you ensure this?**\n\n**Q3: If you ensure this via the learning rate, I think it will potentially lead to higher wall clock convergence time.**\n\n**A (for Q2 and Q3 together):** \nWe do not assume nor ensure \"the magnitude of the weight difference is smaller than the weight\" at all, either in practice or in theoretical analysis. In fact, we use exactly the same learning rate as the non-compression baselines in the experiments.\n\nNote that the statements such as \"magnitudes of weight differences are smaller than those of weights themselves\" in Section 3.1 are simply an intuition and observation on real data that motivates weight difference quantization. It is not an assumption required in the actual theoretical analysis.\n\nWe further explain such an intuition here: For any optimizer, it could be summarized as pesudo code: $w \\leftarrow w - lr * update$, where $w$ is the weight parameter and $lr * update$ is exactly the weight difference. We typically expect that $lr * update$ is much smaller than $w$ itself, otherwise, the optimizer update would override the weight parameter entirely, which doesn't make sense.\n\nAlso, note that we don't really need the assumption of relatively smaller weight differences to prove that weight difference compression is better than weight compression in theory. In fact, the gap between them is whether to converge or not. In Counterexample 4.1 Line 217, we show that under standard settings (smooth function and unbiased gradients) weight compression + SGD may not even converge, no matter how small the learning rate is. In contrast, we provide rigorous proof in Theorem 4.1 (Appendix A) showing that our proposed weight difference compression + SGD converges in the same rate as ordinary SGD.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wqa4Ie8xuO", "reviewer": "Reviewer_4e2h", "summary": "This paper presents a novel scheme for reducing the communication cost of training LLMs using FSDP approach. To this end, the authors suggest to quantize the weights differences (instead of the weights themselves) and applying Hadamard transformations for making the gradients smooth. They show up to ~4x training speedup on 128 GPUs.", "questions": "I have some questions/ideas about the paper. I would be happy if the authors answer them.\n\n1. The paper applies symmetric quantization for all schemes. However, in the communication reduction techniques, it is natural to use asymmetric quantization (which is a bit more accurate). Also, stochastic rounding could be another option to have unbiased gradients. Have you tried those schemes to see the accuracy changes? Maybe you don't need to apply hadamard in that case and the accuracy will be recovered!\n\n2. The main motivation behind the hadamard smoothing is to remove the outliers. This is studied in several works [1, 2, 3, 4] (which are not cited here also). My main question is why we do not see an analysis of the gradient distribution in the paper so we can make sure that the hadamard is important.\n\n3. There isn't any study on the overhead of applying Hadamard scheme. I agree that there are fast hadamard kernels when the input is power-of-two. However, it is not clear for me if your input is not power-of-two. Also, there isn't an ablation on the size of the Hadamard in that case.\n\nReferences:\n\n1. Training Transformers with 4-bit Integers (https://arxiv.org/abs/2306.11987)\n2. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (https://arxiv.org/abs/2404.00456)\n3. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (https://arxiv.org/pdf/2402.04396)\n4. SpinQuant: LLM quantization with learned rotations (https://arxiv.org/abs/2405.16406)", "limitations": "Yes. The author discuss that the work focuses only on communication reduction.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper studies a highly important problem (communication cost of llm training). \n2. Although the ideas of quantizing the weights difference and Hadamard smoothing are not new, using them in this direction is interesting.\n3. The scheme is evaluated on a solid setting (16x4 A100 and 16x8 H100).\n4. The theoretical convergence analysis of difference quantization is nice.\n5. The paper is well-written and easy to follow.", "weaknesses": "1. Although, the idea is interesting, I think the authors should provide more ablations on various aspects of their design choice (see questions section) to elaborate the approach better.\n2. To motivate the problem more, I would like to see some practical analysis of the communication volume based on the number of nodes. For example, how much data (in GB) will be sent between gpus in a single node (and between different nodes) when we scale to a high number of gpus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n**W2: To motivate the problem more, I would like to see some practical analysis of the communication volume based on the number of nodes. For example, how much data ...**\n\nA: The practical communication volume for data parallelism is primarily determined by factors such as model size, the number of GPUs, and the quantization strategy employed. We present the practical volume result for different models below. Note that for larger models (6.7B and above), tensor parallelism is employed, so the intra-node data-parallel size may not equal the number of GPUs within a node.\n\n#### Notation\n| Model Size | GPUs per Node | Nodes | Gradient Quant Group | Weight Quant Group |\n|--|----|--|--|--|\n| M          | g         | N     | G                 | W                    |\n\n#### Number of Bytes sent by a single GPU (when data parallel size = number of total GPUs, g > 1, N > 1)\n\n|  | Baseline &nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp; | SDP4Bit |\n|-----|----|----|\n| **Weight**     |                         |                             |\n| Intra:         | $$2M\\frac{gN - 1}{gN}$$ | $$(0.5M + 4\\lceil M/W\\rceil)\\frac{gN - 1}{gN}$$ |\n| Inter:         | $$2M\\frac{gN - 1}{gN}$$ | $$(0.5M + 4\\lceil M/W\\rceil)\\frac{gN - 1}{gN}$$ |\n| **Gradient**   |                         |                             |\n| Intra:         | $$4M\\frac{gN - 1}{gN}$$ | $$(M + 4\\lceil M/G\\rceil)\\frac{g - 1}{g}$$       |\n| Inter:         | $$4M\\frac{gN - 1}{gN}$$ | $$(0.5\\frac{M}{g} + 4\\lceil\\frac{M}{gG}\\rceil)\\frac{N - 1}{N}g$$ |\n\n#### Number of Bytes sent per GPU per Iteration (32 A100 GPUs)\n\n| Model Size | Baseline Weight         | Baseline Gradient             | SDP4Bit Weight             | SDP4Bit Gradient           |\n|----|------|------|-------|------|\n| GPT-1.3B   | Intra: 2.37 GB, Inter: 2.37 GB | Intra: 4.75 GB, Inter: 4.75 GB | Intra: 0.60 GB, Inter: 0.60 GB | Intra: 0.95 GB, Inter: 1.15 GB |\n| GPT-2.7B   | Intra: 4.79 GB, Inter: 4.79 GB | Intra: 9.57 GB, Inter: 9.57 GB | Intra: 1.20 GB, Inter: 1.20 GB | Intra: 1.91 GB, Inter: 1.15 GB |\n| GPT-6.7B   | Intra: 0 GB, Inter: 2.72 GB | Intra: 0 GB, Inter: 5.45 GB | Intra: 0 GB, Inter: 0.68 GB | Intra: 0 GB, Inter: 0.72 GB |\n| GPT-13B    | Intra: 0 GB, Inter: 2.30 GB | Intra: 0 GB, Inter: 4.61 GB | Intra: 0 GB, Inter: 0.58 GB | Intra: 0 GB, Inter: 0.61 GB |\n| GPT-18B    | Intra: 0 GB, Inter: 3.29 GB | Intra: 0 GB, Inter: 6.58 GB | Intra: 0 GB, Inter: 0.83 GB | Intra: 0 GB, Inter: 0.87 GB |\n\n\nThis data illustrates the communication volume involved in different model sizes and quantization strategies. Notably, larger models like GPT-6.7B and above also enable tensor parallelism which will decrease intra-node dp size, eliminating intra-node dp communication.\n\n### Questions:\n**Q1**\n\n**A:** We appreciate the insightful question regarding the use of asymmetric quantization and stochastic rounding. In our experiments, we've applied symmetric quantization with stochastic rounding across all methods to ensure a fair comparison. Our choice aligns with the baseline papers such as ZeRO++ for consistent comparison.\n\nWhile we acknowledge that different compressors can have varying impacts on performance, our main contributions lie in the areas of weight difference quantization and two-level mixed-precision quantization. The exploration of different compressor types, such as asymmetric quantization or even sparsification and their potential benefits, is orthogonal to our work. Although investigating different compressors is valuable, it's beyond the scope of our current study. Also note that one of our main contributions in gradient quantization is the system optimization (kernel fusion, etc.) which could also be applied to asymmetric quantization.\n\n**Q2**\n\n**A:** We appreciate the suggestion to analyze gradient distribution to justify the use of the Hadamard Transform. We've added and additional analysis illustrated in **Figure 3 of our rebuttal PDF** for the gradient distribution. This analysis demonstrates the necessity of the Hadamard Transform in handling gradient outliers, thereby justifying its use in our methodology. We will also include the additional relevant works ([2, 3, 4]) to our revised manuscript, as well as the analysis of gradient distribution.\n\n**Q3**\n\n**A:** The overhead reduction is two-fold: the choice of a small Hadamard matrix size and kernel fusion. The experiments in Table 4 (TLq-HS vs. Int4-Int4) demonstrate that there is nearly no additional overhead due to these optimizations.\n\nAs described in Section 3.3, our proposed TLq-HS uses a Hadamard matrix size of 32x32, which we consider the best trade-off between accuracy and performance. Although larger Hadamard matrices can better smooth outliers, the improvement is modest (**illustrated in the table below with different Hadamard size**). Additionally, larger matrices increase computational complexity, which can become a bottleneck, especially on older GPU versions, and can result in inefficient memory access patterns due to larger tiling sizes.\n\nGiven the small size of the Hadamard matrix (32x32), the Hadamard transformation is primarily a memory-bound operation on contemporary GPUs. This characteristic allows us to seamlessly integrate the Hadamard transform with the quantization kernel, resulting in nearly zero additional overhead. To further illustrate the impact of the Hadamard transformation on performance, we provide (de)quantization throughput experiment in our **rebuttal pdf table 2**, which is tested on an A100 GPU. \n\nIf the input size is not divisible by 32, we pad the input with zero accordingly.\n\nAs requested, the ablation study of **validation loss with different Hadamard sizes** is provided below. We will also include these results in the revised version. \n|  GPT-125M   | Val Loss  |\n|-|-|\n| Baseline | 2.29392 |\n| TLq-HS (Hadamard Size=32)  | 2.29528|\n| TLq-HS (Hadamard Size=64)  | 2.29597|\n| TLq-HS (Hadamard Size=128) | 2.29612 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' critical assessment of our work. Below, we provide the relevant figures and results to address the questions and concerns raised.

**Contents in Rebuttal PDF**

1. Table of notations that explains the abbrevations of algorithms such as W4, Int4-Int4, TLq, etc.
2. Comparison of Accuracy versus Wall-Clock Time Results
3. Additional ablation Study: Direct comparison between our proposed Int8-Int4 gradient quantization (denoted as TLq) and the original ZeRO++ gradient quantization (denoted as Int4-Int4)
4. Analysis of Gradient Distribution Before and After Hadamard Transformation
5. Comparison of (De)Quantization Kernel Speed with and without Hadamard Transformation

Meta-review: All reviewers have given favorable ratings for this paper, ranging from 5 to 7.
Average Rating: 6.00 (Min: 5, Max: 7)
The following strengths of the paper have been highlighted. However, as there are few strong recommendations for this paper, an acceptance as a poster presentation is considered appropriate.

1: This paper proposes an innovative approach to reduce communication overhead in large language model (LLM) training with Sharded Data Parallelism (ShardedDP). The proposed method introduces quantization based on weight differences and two-level smooth quantization, compressing communication to nearly 4 bits without significantly compromising accuracy.

2: The proposed method demonstrates up to a 4.08x increase in end-to-end throughput when training GPT models with up to 6.7 billion parameters. Additionally, theoretical convergence guarantees are provided, and the impact on training loss is confirmed to be minimal.

3: The proposed method involves a system and algorithm co-design that includes GPU kernel implementation and runtime optimization, minimizing computation overhead. This aspect is highly valued as it contributes not only theoretical ideas but also practical implementation and optimization.

4: The method has been evaluated at a scale of up to 128 GPUs and is applicable to models of varying sizes. By improving the efficiency of communication compression, the approach proves useful in a wide range of training scenarios.
Decision: Accept (poster)