SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training

Authors: jinda  jia
cong  xie
hanlin  lu
daoce  wang
hao  feng
chengming  zhang
baixi  sun
haibin  lin
zhi  zhang
xin  liu
dingwen  tao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 19:40:20

Abstract: Recent years have witnessed a clear trend towards language models with an ever-increasing number of parameters, as well as the growing training overhead and memory usage. Distributed training, particularly through Sharded Data Parallelism (ShardedDP) which partitions optimizer states among workers, has emerged as a crucial technique to mitigate training time and memory usage. Yet, a major challenge in the scalability of ShardedDP is the intensive communication of weights and gradients. While compression techniques can alleviate this issue, they often result in worse accuracy. Driven by this limitation, we propose SDP4Bit (Toward 4Bit Communication Quantization in Sharded Data Parallelism for LLM Training), which effectively reduces the communication of weights and gradients to nearly 4 bits via two novel techniques: quantization on weight differences, and two-level gradient smooth quantization. Furthermore, SDP4Bit presents an algorithm-system co-design with runtime optimization to minimize the computation overhead of compression. Additional to the theoretical guarantees of convergence, we empirically evaluate the accuracy of SDP4Bit on the pre-training of GPT models with up to 6.7 billion parameters, and the results demonstrate a negligible impact on training loss. Furthermore, speed experiments show that SDP4Bit achieves up to 4.08× speedup in end-to-end throughput on a scale of 128 GPUs.

Main paper: 
 introduction: 
 Large Language Models (LLMs) are increasingly utilized across various applications, leading to a trend toward larger model sizes. This expansion in model size significantly escalates training overheads, making the process more costly and resource-intensive. To mitigate the time-consuming nature of training LLMs, it is common to employ multiple GPUs in a data-parallel configuration. However, naive Data Parallelism (DP) necessitates that each GPU replicates the entire optimizer states, a strategy often impractical due to the limited memory capacity of individual GPUs. This limitation becomes particularly critical with the substantial size of modern LLMs. Sharded Data Parallelism (ShardedDP) evolves from naive DP to reduce the memory footprint by sharding optimizer states among GPUs. However, the sharding mechanism significantly changes the communication pattern of DP, which brings up new challenges in system optimization. As a result, ShardedDP suffers from heavy communication overheads of both weights and gradients, particularly when internode bandwidth is limited. This can significantly increase the end-to-end (E2E) training time, especially when using a small gradient accumulation step.Quantization is a widely used strategy to reduce the communication overhead of naive DP, albeit with some accuracy loss. Unfortunately, few prior studies have specifically addressed the issue of communication reduction in ShardedDP. Recently, QSDP [18] and ZeRO++ [32] attempted to quantize the communication of ShardedDP to Int4. However, when pushing the communication ratio to its limits, both QSDP and ZeRO++ fail to maintain comparable training loss to the baseline. Furthermore, ZeRO++ lacks theoretical convergence guarantees, and QSDP is limited to one specific quantizer called "random shift" and strong assumptions. Thus, there is no effective solution to reduce ShardedDP's communication to nearly 4 bits without compromising the training loss.To address these issues, this paper proposes a novel communication reduction strategy, SDP4Bit. SDP4Bit comprises two main techniques: (1) Quantization on Weight Differences: Instead of directly quantizing weights, we apply 4-bit quantization to compress the weight differences between current and previous iterations; (2) Two-Level Gradient Smooth Quantization: We apply 8-bit quantization to intra-node gradients and 4-bit quantization to inter-node gradients, with Hadamard Transform for smoothing the outliers. To the best of our knowledge, SDP4Bit is the first work to successfully reduce both gradients and weights to nearly 4 bits without compromising training accuracy. As shown in Figure 1 , the training validation loss for GPT-6.7B using SDP4Bit is closely aligned with full precision training. Our main contributions are summarized as follows:• We propose a low-bit (i.e., nearly 4-bit) communication reduction strategy for ShardedDP that preserves E2E training accuracy. • We establish a convergence guarantee for the proposed strategy, showing the same convergence rate as the ordinary Stochastic Gradient Descent (SGD), with extended choices of biased compressors and weaker assumptions compared to the previous theoretical results. • We implement our method within the Megatron-LM framework and enhance it with runtime optimizations such as buffer reuse, operation pruning, and kernel fusion. • Our results validate that SDP4Bit successfully compresses the communication of weights and gradients to nearly 4 bits, with a negligible impact on final loss. Notably, compared to non-quantized baseline, it achieves 4.08× speedup for a GPT-18B model trained on 128 H800 GPUs. 
sharded data parallelism: 
 Sharded Data Parallelism (ShardedDP) modifies traditional Data Parallelism (DP) to reduce the memory footprint per GPU. Unlike traditional DP, which duplicates high-precision optimizer states (typically including model weights and momentum variables in Float32) on each GPU, ShardedDP partitions them across all GPUs. Each GPU manages 1 P of the optimizer states, hence reducing the corresponding memory footprint by a factor of 1 P , where P represents the number of GPUs involved. With high-precision model weights sharded across GPUs (referred to as "main weights"), an allgather operation is required to collect the weights for the forward-backward steps (typically in relatively low precision, such as Float16, referred to as "model weights"). For gradient synchronization, a reduce-scatter operation is performed before the optimization steps to ensure that each GPU has the corresponding shard of averaged gradients. In summary, each iteration necessitates an all-gather for weights and a reduce-scatter for gradients.Driven by the need to train larger models within the constraints of GPU memory, ShardedDP is incorporated into several popular training frameworks, including Megatron-LM (Distributed Optimizer), DeepSpeed (ZeRO), and PyTorch (FSDP), each with slightly different implementation strategies. Notably, ZeRO-3 and FSDP release the collected weights after each computation to enhance memory efficiency, necessitating additional weight collective communication during the backward pass. Conversely, ZeRO-2 and Megatron-LM retain the collected weights throughout, thus eliminating the need for weight collection during the backward pass. Our weight reduction strategy is particularly well-suited for Megatron-LM, as it maintains a full model's weights at all times (see Algorithm 2). Additionally, Megatron-LM provides flexible parallelism support, such as tensor parallelism, which partitions models vertically to alleviate memory limitations. This approach enables the training of larger models compared to DeepSpeed.Quantization is a commonly used strategy in data compression. In this paper, we explore symmetric linear (integer) quantization due to its low overhead and latency. It is defined as follows:xint = round x s • (2 k-1 -1) , s = max(x),where k represents the bit-width of the quantized values, and s is referred to as "scales".Additionally, group-wise quantization [25] is employed to minimize quantization error by dividing the data into multiple groups and quantizing each group individually. This approach results in a lower compression ratio due to the need to store additional scales.State-of-the-art (SOTA) collective communication libraries (e.g., NCCL, Gloo) employ a ring-based algorithm for its optimal bandwidth [21] . This algorithm executes reduce-scatter operations across P -1 rounds, during which each GPU sends local data and aggregates the received data. When quantization is applied, this necessitates P -1 rounds of quantization and dequantization, potentially leading to error propagation and increased latency [11] . Some strategies replace reduce-scatter with all-to-all communication, but this increases inter-node communication, typically with lower bandwidth.ZeRO++ [32] modifies this approach by substituting the conventional reduce-scatter (used by QSDP) with two all-to-all operations (shown in Algorithm 1, with different colors to distinguish ZeRO++ from QSDP). The first operation is confined within each node, and post-reduction, the data size is diminished to 1 N , where N is the number of GPUs per node. The subsequent all-to-all operation occurs between GPUs across different nodes that share the same local rank. In ZeRO++, each all-to-all operation follows a 4-bit quantization step to minimize communication data size.As shown in Figure 5 , while this method efficiently integrates quantization into reduce-scatter without augmenting inter-node communication, the repeated 4-bit quantization steps can accumulate quantization errors, potentially leading to suboptimal training outcomes. Step 1:Intra-node all-to-all (Int8)Step 2:Inter-node all-to-all (Int4) As discussed in Section 2.1, ShardedDP requires each GPU to send/receive updated weights (main weights) to/from other GPUs in each iteration. However, weights generally exhibit a wide range and directly applying 4-bit quantization leads to significant quantization errors. Even with groupwise quantization, a gap in E2E training loss compared to full precision remains despite using small group sizes.GPU 0 GPU 1 Node 0 GPU 2 GPU 3 GPU 0 GPU 1 Node 1 GPU 2 GPU 3 GPU 0 GPU 1 Node 2 GPU 2 GPU 3 GPU 0 GPU 1 Node 3 GPU 2 GPU 3qWD: To address this issue, we quantize weight differences instead of the original weights during communication. As illustrated in Figure 2 and Algorithm 2, after the optimizer step, each GPU calculates the differences between the main weights and the model weights. These differences are then quantized and all-gathered across all GPUs. After all-gathering, each GPU dequantizes the received data to obtain the weight differences. These differences are then added to the model weights to obtain the updated weights.There are two main benefits to applying quantization to weight differences.In practice, weight differences are generally easier to quantize. As shown in Figure 4 , weight differences are more uniformly distributed in a smaller range compared to the weights themselves, resulting in smaller errors for INT4 quantization. Furthermore, since intuitively the magnitudes of weight differences are smaller than those of weights themselves (informally supposed that ∥δw t ∥ = ∥w t -w t-1 ∥ < ∥w t ∥) and the relative quantization errors are similar between weights and weight differences (informally supposed that ∥q(δwt)-δwt∥ ∥δwt∥ ≈ ∥q(wt)-wt∥ ∥wt∥ ), the weight differences compression potentially has a smaller error relative to the weights themselves:∥q(δwt)-δwt∥ ∥wt∥ ⪅ ∥q(wt)-wt∥ ∥wt∥ , where q(•) is the quantization function.2) In theory, weight differences compression improves convergence compared to naive weight compression. When extended to biased compressors, we present theoretical guarantees for convergence at the same rate as ordinary SGD, as detailed in Section 4.2. In contrast, we demonstrate that using biased compressors directly on weights can lead to convergence failure, as illustrated in an example in Section 4.1. This proves that biased compressors are not compatible with QSDP or ZeRO++.3.2 Two-Level Gradient Smooth Quantization 3.2.1 Two-level Gradient Quantization (TLq)As discussed in Section 2.3, the 2-step all-to-all communication strategy benefits gradient communication when quantization is applied. However, it also introduces error accumulation due to consecutive 4-bit quantization steps, necessitating additional rounds of training and communication that diminish the per-iteration communication savings. We observe that applying ULq, which quantizes gradeints to extremely low precision, such as 4-bit, leads to noticeable deviations in training loss compared to full-precision training, as illustrated in Figure 5 .TLq: Instead of employing a global 4-bit quantization strategy for both inter-node and intra-node all-to-all communications, we propose a two-level precision quantization strategy. This approach balances performance and accuracy by enhancing accuracy without introducing additional overhead.For intra-node all-to-all communication, gradients are quantized to INT8 before sending. After receiving the data, each GPU dequantizes the received data back to the full precision (i.e., FP32) for local reduction. This reduced data is then quantized to INT4 to minimize inter-node communication overhead. The detailed methodology is depicted in Figure 3 . Since the two all-to-all operations utilize different network bandwidths, their communications can be effectively overlapped (see Table 4 ). 
tlq with hadamard smoother (tlq-hs): 
 While TLq brings the training loss closer to the baseline, it does not achieve perfect alignment. In gradient quantization, outliers can significantly amplify quantization errors. Although group-wise quantization isolates outliers to minimize their impact on the precision of values in other groups, the values within the same group remain affected.TLq-HS: To mitigate the outlier issue, we apply Hadamard transform [9] to the gradients before quantization. The Hadamard transform, a specific type of generalized Fourier transform, exhibits properties such as H = H T and H • H T = I, distributing outlier information across nearby elements and effectively smoothing them out. For a detailed description of the methodology, see Algorithm 3. ĝ ← Hadamard(grad)3:qg 8bit ← Quantize8Bit(ĝ) 4: list( qg 8bit_intra ) ← IntraAlltoAll( qg 8bit ) 5:list(ĝ intra ) ← Dequantize(list( qg 8bit_intra ))6:list(ĝ intra ) ← Hadamard(list(ĝ intra ))7:ĝreduced ← Reduction(list(ĝ intra ))8:ĝreduced ← Hadamard(ĝ reduced )9:qg 4bit ← Quantize4Bit(ĝ reduced ) 10: list( qg 4bit_inter ) ← InterAlltoAll( qg 4bit ) 11:list(ĝ inter ) ← Dequantize(list( qg 4bit_inter ))12:ĝfinal_reduced ← Reduction(list(ĝ inter ))13:g f inal_reduced ← Hadamard(ĝ f inal_reduced )Optimizing Memory Efficiency for Weight Differences Computation: After the all-gather communication, each GPU receives weight differences from the others, which are then added to the model weights to update them. As discussed in Section 2.1, with ShardedDP enabled in Megatron-LM, each GPU maintains a complete copy of the model weights for forward and backward computation. This contrasts with ZeRO, where model weights are released after computation. By reusing these locally stored model weights in Megatron-LM, our implementation eliminates the need for additional buffers to retain model weights for calculating the differences, thus enhancing memory efficiency.Simplifying Hadamard Transforms: In a naive implementation, the Hadamard transform would be applied at each step before quantization and after dequantization. However, by leveraging the orthogonality of the Hadamard transform, i.e. H • H = I, we omit the transform after the intra-node all-to-all dequantization (Algorithm 3, Line 6) and before the inter-node quantization (Algorithm 3, Line 8). Furthermore, by utilizing the distributive property, i.e., i Hg i = H i g i , we move the second Hadamard transform from after the inter-node dequantization (Algorithm 3, Line 11) to after the final reduction (Algorithm 3, Line 12). These simplifications reduce the unnecessary computational overhead associated with repeated Hadamard transforms.Fusing GPU Kernels with Group Size Alignment: To mitigate additional data movement from global memory-which typically exhibits the slowest memory bandwidth-, we fuse the Hadamard transform with the (de)quantization operations into a single CUDA kernel. This fusion allows the operations to run nearly as fast as the quantization operation alone. It is worth noting that, for this fusion to be efficient, there must be an alignment between the two. Specifically, the size of the quantization group must be divisible by the size of the Hadamard matrix, ensuring that memory traffic remains within the kernel block. We choose H to be small (e.g., 32×32) because, at this size, the transform operation on the GPU is typically memory-bound and incurs minimal overhead.While larger H sizes offer better smoothing capabilities, we find that a 32×32 matrix is sufficient to effectively smooth outliers in gradients.4 Theoretical Analysis 
counterexample of biased weight compression: 
 One of the advantages of our proposed weight difference compression is the compatibility to both biased and unbiased compressors. Note that using biased compressors directly on weight compression incurs issues in convergence under standard assumptions, as avoided in QSDP [18] or ZeRO++ [32] .We illustrate such issues in the following toy example.Counterexample 4.1. Consider a least square problem with w * = (0, 0) ⊤ : min w∈R 2 f (w) = ∥w∥ 2 , and stochastic gradient g(w) = (4w 1 , 0) ⊤ with probability 0.5 and g(w) = (0, 4w 2 ) ⊤ with probability 0.5, thus E[g(w)] = ∇ w f (w). We use the initial value w init = (1, -1) ⊤ , the learning rate η < 0.125, and the following nearest ternary quantizer: s = max(|w|), q(w) = round(w/s) * s, where | • | is element-wise absolute value, and round(•) quantizes each element to the nearest value in {-1, 0, 1}. It is easy to check that for SGD under such settings, the weights before quantization will be either (1 -4η, -1) ⊤ or (1, -1 + 4η) ⊤ , resulting in q(w) = (1, -1) ⊤ , which means that SGD with ternary weight quantization gets stuck at the initial value in this case, while SGD without weight quantization and SGD with weight difference quantization both converge to the optimal.To theoretically analyze the convergence of our distributed training algorithm with communication compression, we focus on the following SGD variant with gradient compression and weight difference compression. We use SGD to solve the following optimization problem: Compute gradient:f * = min w f (w), where f (w) is the objective function, w ∈ R d is the model parameter.g t-1 = ∇f ( wt-1 ; ζ t-1 ) 5:Compress gradient: gt-1 = U g (g t-1 )Update main weights: w t ← w t-1 -ηg t-1Compress weight difference: ∆t = C w (w t -wt-1 )Update compressed weights: wt ← wt-1 + ∆t 9: end for Note that we use unbiased compressors for gradient reduction, and arbitrary (potentially biased) compressors for weight collection. We formally define these two classes of compressors as follows. Definition 4.1 (Unbiased κ-approximate compressor [1] ). An operator [13] ). An operatorU : R d → R d is a κ- approximate compressor for κ ≥ 0 if E[U(v)] = v and E∥U(v) -v∥ 2 ≤ κ∥v∥ 2 , ∀v ∈ R d . Definition 4.2 (δ-approximate compressorC : R d → R d is a δ-approximate compressor for δ ∈ [0, 1] if E∥C(v) -v∥ 2 ≤ (1 -δ)∥v∥ 2 , ∀v ∈ R d . Remark 4.1.Note that, in a certain sense, the class of δ-approximate compressors contains the class of unbiased compressors. It is easy to check that any κ-approximate unbiased compressor U can be converted to a 1 1+κ -approximate biased compressor C(v) = 1 1+κ U(v). Furthermore, the class of δ-approximate compressors typically provides more options such as top-k sparsifiers, and top-k low-rank compressors. Thus, we consider arbitrary (biased or unbiased) δ-approximate compressors for weight compression in our theoretical analysis. Remark 4.2. For distributed training with P workers, we define the compressed gradient as gt = U g (g t ) = 1P i∈[P ] U ′ g (g t,i), where g t = 1P i∈[P ] g t,i , and g t,i is the stochastic gradient from the ith worker in t iteration. We assume that U g is an unbiased κ-approximate compressor of the average gradient g t . Assumption 4.1. (Smoothness) We assume that f (x) is L-smooth: ∥∇f (x) -∇f (y)∥ ≤ L∥x - y∥, ∀x, y ∈ R d , which implies f (y) -f (x) ≤ ⟨∇f (x), y -x⟩ + L 2 ∥y; ζ) - ∇f (w)∥ 2 |w] ≤ ρ∥∇f (w)∥ 2 + σ 2 ([27], Assumption 3).We derive the following error bounds on the convergence of SDP4Bit under the above assumptions. All proofs can be found in Appendix A. T t=0 E[∥∇f ( wt)∥ 2 ] T + 1 ≤ 80L 2 δ + ρκ + ρ + κ (f (w0) -f * ) T + 1 + 4σ (11 -δ)(κ + 1)L(f (w0) -f * ) T + 1 .Remark 4.3. Note that compared to QSDP [18] , our convergence analysis does not require Polyak-Łojasiewicz condition or the specific choice of weight quantization (random shift). In other words, Theorem 4.1 shows that our proposed algorithm has the same O 1 √ T convergence rate as ordinary SGD for general non-convex functions, but under much weaker assumptions compared to QSDP. 
experimental setup: 
 Hardware: The experiments are conducted on two different clusters to evaluate SDP4Bit across varying network environments: 1) 16 nodes, each node equipped with 4 Nvidia A100-SXM4-40GB GPUs. All nodes are interconnected with a 100 Gbps Slingshot10 network, providing slower internode bandwidth. 2) 16 nodes, each node equipped with 8 Nvidia H800-SXM5-80GB GPUs. Each node is connected using 8 InfiniBand links, achieving a total bandwidth of 3.2 Tbps, providing higher inter-node bandwidth.Baselines: We use BFloat16/Float32 (weights/gradients) mixed-precision in Megatron-LM [26] as our basic Baseline for both accuracy and E2E throughput analysis. Within each set of experiments, we ensure consistent hyper-parameters to ensure fairness. Detailed parameters are provided in Appendix D. Additionally, we implement another baseline for comparison in Megatron-LM, using the same quantization strategy in ZeRO++, employing 4-bit quantization for both weights (groupwise weight quantization, refered to as qW) and gradients (twice all-to-all with uniform level 4-bit quantization, refer to as ULq).Dataset and Models: To demonstrate that SDP4Bit does not adversely affect end-to-end training loss, we conduct pre-training on GPT-series [23] models ranging from 125M to 6.7B parameters on the Pile dataset [8] , using validation loss as the accuracy measure. Each test runs for 80,000 iterations, processing over 40 billion tokens. For throughput evaluation, we select models ranging from 1.3B to 18B parameters, with end-to-end training throughput as the metric. In these tests, the accumulation step is set to 1. Note that model parallel is required for models larger than 6.7B, and different tensor parallel sizes are used on A100 and H800 clusters for models larger than 13B. Please refer to Appendix B Table 8 for detailed model parallel configuration.First, we analyze the impact of SDP4Bit on the accuracy of E2E training. As shown in Table 1 , the training results for three different model sizes indicate that the final loss of SDP4Bit is comparable to the baseline, with a maximum increase of only 0.24%. Additionally, Figure 1 details the training curve of GPT-6.7B, demonstrating that the training curve of SDP4Bit perfectly aligns with the baseline. This indicates that the impact of SDP4Bit on accuracy is negligible. In contrast, the 4-bit quantization strategy in ZeRO++ (which directly applies quantization to weights and uniformly uses 4-bit quantization with all-to-all for gradients) results in significant accuracy degradation.Next, we break down and analyze each strategy within SDP4Bit. 1) For weight communication reduction, as shown in Table 1 , directly quantizing weights (qW) to 4 bits results in a validation loss that increase of up to 12% compared to the baseline. In contrast, our weight difference quantization (qWD) method achieves a validation loss nearly identical to the baseline. Notably, we use a consistent quantization group size of 2048 for both tests. 2) For gradient communication reduction, as shown in Figure 5 , applying Uniform Level quantization (ULq), similar to the method used in ZeRO++, results in a significant gap in the loss compared to the baseline. In comparison, our Two Level quantization (TLq) siginificantly mitigate the loss gap between with baseline. Additionally, Figure 6 illustrates the effectiveness of the Hadamard transformation in smoothing outliers. After Hadamard Transform In addition, we demonstrate the scalability of SDP4Bit using GPT models of 6.7B and 13B parameters, with tests conducted on up to 128 GPUs, as shown in Figure 7 . Under low bandwidth conditions, SDP4Bit achieves an average speedup of 3.40× for the 6.7B model and 2.49× for the 13B model. In high-bandwidth InfiniBand environments, the speedup averages 3.08× for the 6.7B model and 3.73× for the 13B model. The comparatively lower speedup for the 13B model under low bandwidth conditions can be attributed to the introduction of pipeline parallelism, which diminishes the proportion of communication handled by ShardedDP. Overall, SDP4Bit consistently maintains stable speedup performance across various GPU numbers and network environments.Components Breakdown. Figure 8 demonstrates the throughput improvement of qWD, TLq-HS, and their combination (SDP4Bit) on two different platforms. qWD alone provides a speedup ranging from 1.1× to 1.2×, while TLq-HS alone results in an E2E speedup of 1.4× to 1.8×. The notable benefit from gradient quantization stems from the high communication overhead associated with Float32 gradients in baseline training, which is higher compared to BFloat16 weights. When they are applied together, SDP4Bit achieves a more substantial speedup, ranging from 1.6× to 2.4×.TLq-HS vs. ULq. Table 4 compares gradient quantization between TLq-HS and ULq. The results show that although TLq-HS employs 8-bit quantization for intra-node gradient communication, it introduces negligible overhead compared to 4-bit communication. This is due to 1) the high bandwidth of intra-node communication and 2) the fact that most intra-node communication is overlapped with the slower inter-node communication.Hadamard Kernel Fusion. Table 4 shows that, compared to the SDP4Bit without fusing Hadamard Transform kernel, our optimized SDP4Bit reduces gradient communication overhead by 29%. Additionally, we provide a throughput comparison in Table 5 to further illustrate the impact of the Hadamard transformation. The results confirm that our Hadamard kernel fusion effectively reduces the overhead, making the transformation nearly zero-overhead and even matching the performance of quantization without the Hadamard transformation.Convergence with Different Group Sizes. Table 3 examines the impact of various quantization granularities on the end-to-end validation loss during the pre-training of the GPT-125M model. For TLq-HS, a gradient quantization group size of 128 presents sufficient, with smaller sizes yielding no significant accuracy improvements. For qWD, a quantization group size of 2048 achieves training accuracy comparable to the baseline. Table 3 also presents the 4-bit weight quantization (qW) while using small group size. It is evident that even with very small group size (e.g., 32), direct 4-bit quantization leads to a significant gap in accuracy compared to the baseline, making 4-bit quantization for weights suboptimal. 
related work: 
 Apart from ZeRO++ [32] and QSDP [18] , which are specifically designed for communication compression in ShardedDP, most previous studies have focused on traditional DP, primarily utilizing gradient compression. This includes both unbiased compression techniques [1, 33, 38, 5] , which employ randomized compressors, and biased compression methods with error compensation [12, 31, 30, 29, 24] that require extra storage for residual errors, making them less suitable for resourceintensive training of LLMs. Other strategies like local optimization or federated learning reduce communication frequency rather than volume [16, 28, 35, 34, 2, 20] , but increase memory use, complicating their application in LLM training. In addition, techniques like low-precision training [19, 22] and parameter-efficient fine-tuning [10, 3, 14] minimize the volume of trainable variables to reduce communication. In a different vein, weight quantization for inference has also been explored [7, 6, 39, 37, 4] , employing more resource-intensive methods compared to those used in training to fine-tune compression parameters.The Hadamard transform has been applied to machine learning data, as seen in HQ-MM's [36] compression of activations and THC's [15] gradient communication within a parameter server framework. Unlike THC, SDP4Bit enhances collective communication operations and GPU optimization. 
conclusion: 
 In this paper, we propose SDP4Bit, a communication reduction strategy for Sharded Data Parallelism. SDP4Bit reduces both weight and gradient communication to nearly 4 bits while maintaining model accuracy comparable to the baseline. We implemented SDP4Bit in Megatron-LM and optimized it to reduce quantization overhead. Specifically, our experimental results demonstrate a training speedup of up to 4.08 × on 128 GPUs. This paper focuses on LLM pre-training, but we plan to extend our work to other models and areas such as MoE, computer vision, and fine-tuning in the future. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper does not release any new dataset or new model. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited the original paper that produced the code package or dataset, and included the license in the source code package. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We have included the license in the source code package, which is aligned with the code packages that our work is based on. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We use the following lemma (simplified from [27] , Lemma 14) without proof.Lemma A.1. For every non-negative sequence {r t } t≥0 and any parameters a ≥ 0, c ≥ 0, T ≥ 0, there exists a constant η ≤ 1 a , such that 1 T + 1 T t=0 r t -r t+1 η + cη = 1 T + 1 r 0 -r T +1 η + cη ≤ ar 0 T + 1 + 2 √ cr 0 √ T + 1 .T t=0 E[∥∇f ( wt)∥ 2 ] T + 1 ≤ 80L 2 δ + ρκ + ρ + κ (f (w0) -f * ) T + 1 + 4σ (11 -δ)(κ + 1)L(f (w0) -f * ) T + 1 .Proof. By using smoothness (Assumption 4.1), we havef (w t+1 ) ≤ f (w t ) -η ⟨∇f (w t ), U g (g t )⟩ + η 2 L 2 ∥U g (g t )∥ 2 .Taking expectation w.r.t. the random compressor U g , we haveE gc [f (w t+1 )] ≤ f (w t ) -η ⟨∇f (w t ), g t ⟩ + η 2 L 2 E gc ∥U g (g t )∥ 2 = f (w t ) -η ⟨∇f (w t ), g t ⟩ + η 2 L 2 [∥g t ∥ 2 + E gc ∥U g (g t ) -g t ∥ 2 ] ≤ f (w t ) -η ⟨∇f (w t ), g t ⟩ + η 2 L(κ + 1) 2 ∥g t ∥ 2 .Conditional on w t , taking expectation on the random sample ζ t , we haveE ζ [E gc [f (w t+1 )]] ≤ f (w t ) -η ⟨∇f (w t ), ∇f ( wt )⟩ + η 2 L(κ + 1) 2 E ζ ∥g t ∥ 2 = f (w t ) -η ⟨∇f (w t ), ∇f ( wt )⟩ + η 2 L(κ + 1) 2 E ζ ∥g t -∇f ( wt ) + ∇f ( wt )∥ 2 = f (w t ) -η ⟨∇f (w t ), ∇f ( wt )⟩ + η 2 L(κ + 1) 2 E ζ [∥g t -∇f ( wt )∥ 2 + ∥∇f ( wt )∥ 2 ] ≤ f (w t ) -η ⟨∇f (w t ), ∇f ( wt )⟩ + η 2 L(κ + 1)(ρ + 1) 2 ∥∇f ( wt )∥ 2 + η 2 L(κ + 1)σ 2 2 ≤ f (w t ) -η ⟨∇f (w t ), ∇f ( wt )⟩ + η 2 L(κ + 1)(ρ + 1) 2 ∥∇f ( wt )∥ 2 + η 2 L(κ + 1)σ 2 2 = f (w t ) -η ⟨∇f (w t ) -∇f ( wt ) + ∇f ( wt ), ∇f ( wt )⟩ + η 2 L(κ + 1)(ρ + 1) 2 ∥∇f ( wt )∥ 2 + η 2 L(κ + 1)σ 2 2 = f (w t ) -η ⟨∇f (w t ) -∇f ( wt ), ∇f ( wt )⟩ -η∥∇f ( wt )∥ 2 + η 2 L(κ + 1)(ρ + 1) 2 ∥∇f ( wt )∥ 2 + η 2 L(κ + 1)σ 2 2 = f (w t ) -η 1 - ηL(κ + 1)(ρ + 1) 2 ∥∇f ( wt )∥ 2 -η ⟨∇f (w t ) -∇f ( wt ), ∇f ( wt )⟩ + η 2 L(κ + 1)σ 2 2 ≤ f (w t ) -η 1 - ηL(κ + 1)(ρ + 1) 2 ∥∇f ( wt )∥ 2 + η 2 ∥∇f (w t ) -∇f ( wt )∥ 2 + η 2 ∥∇f ( wt )∥ 2 + η 2 L(κ + 1)σ 2 2 ▷ ⟨a, b⟩ ≤ 1 2 ∥a∥ 2 + 1 2 ∥b∥ 2 = f (w t ) - η 2 [1 -ηL(κ + 1)(ρ + 1)] ∥∇f ( wt )∥ 2 + η 2 ∥∇f (w t ) -∇f ( wt )∥ 2 + η 2 L(κ + 1)σ 2 2 .Again using smoothness, and taking η ≤1 2L(ρ+1)(κ+1) , we have -η 2 [1 -ηL(κ + 1)(ρ + 1)] ≤ -η 4, and, we haveE ζ [E gc [f (w t+1 )]] ≤ f (w t ) - η 2 [1 -ηL(κ + 1)(ρ + 1)] ∥∇f ( wt )∥ 2 + ηL 2 2 ∥w t -wt ∥ 2 + η 2 L(κ + 1)σ 2 2 ≤ f (w t ) - η 2 [1 -ηL(κ + 1)(ρ + 1)] ∥∇f ( wt )∥ 2 + ηL 2 2 ∥e t ∥ 2 + η 2 L(κ + 1)σ 2 2 ≤ f (w t ) - η 4 ∥∇f ( wt )∥ 2 + ηL 2 2 ∥e t ∥ 2 + η 2 L(κ + 1)σ 2 2 ,where we define the sequence e t = w t -wt , e 0 = 0. Now we establish the upper bound of the sequence ∥e t ∥ 2 as follows.First, using w t+1 = w t -ηU g (g t ) and wt+1 = wt +C w (w t+1 -wt ), we have the following equations:w t+1 -wt+1 = e t+1 = w t -wt -ηU g (g t ) -C w (w t+1 -wt ) = e t -ηU g (g t ) -C w (e t -ηU g (g t ))Taking expectation w.r.t. the random compressor C w , we haveE wc [∥e t+1 ∥ 2 ] = E wc [∥e t -ηU g (g t ) -C w (e t -ηU g (g t ))∥ 2 ] ≤ (1 -δ)∥e t -ηU g (g t )∥ 2 .Taking expectation w.r.t. the random compressor U g , we haveE gc [E wc [∥e t+1 ∥ 2 ]] ≤ (1 -δ)E gc [∥e t -ηU g (g t )∥ 2 ] = (1 -δ)E gc [∥e t -ηg t + ηg t -ηU g (g t )∥ 2 ] = (1 -δ)∥e t -ηg t ∥ 2 + (1 -δ)η 2 E gc [∥g t -U g (g t )∥ 2 ] ≤ (1 -δ)∥e t -ηg t ∥ 2 + (1 -δ)η 2 κ∥g t ∥ 2 .Conditional on w t , taking expectation on the random sample ζ t , we haveE ζ [E gc [E wc [∥e t+1 ∥ 2 ]]] ≤ (1 -δ)E ζ [∥e t -η∇f ( wt ) + η∇f ( wt ) -ηg t ∥ 2 ] + (1 -δ)η 2 κE ζ [∥g t -∇f ( wt ) + ∇f ( wt )∥ 2 ] = (1 -δ)∥e t -η∇f ( wt )∥ 2 + (1 -δ)(κ + 1)η 2 E ζ [∥g t -∇f ( wt )∥ 2 ] + (1 -δ)η 2 κ∥∇f ( wt )∥ 2 ≤ (1 -δ)∥e t -η∇f ( wt )∥ 2 + (1 -δ)(κ + 1)η 2 (ρ∥∇f ( wt )∥ 2 + σ 2 ) + (1 -δ)η 2 κ∥∇f ( wt )∥ 2 = (1 -δ)∥e t -η∇f ( wt )∥ 2 + (1 -δ)η 2 (ρκ + ρ + κ)∥∇f ( wt )∥ 2 + (1 -δ)(κ + 1)η 2 σ 2 .With ∀b > 0, we haveE ζ [E gc [E wc [∥e t+1 ∥ 2 ]]] ≤ (1 -δ)(1 + b)∥e t ∥ 2 + (1 -δ)(1 + b -1 )∥η∇f ( wt )∥ 2 + (1 -δ)η 2 (ρκ + ρ + κ)∥∇f ( wt )∥ 2 + (1 -δ)(κ + 1)η 2 σ 2 = (1 -δ)(1 + b)∥e t ∥ 2 + (1 -δ)η 2 [1 + b -1 + (ρκ + ρ + κ)]∥∇f ( wt )∥ 2 + (1 -δ)(κ + 1)η 2 σ 2 .Then, by taking b = δ 2(1-δ) , we have(1 -δ)(1 + b) = 1 -δ 2 , 1 + b -1 = 2-δ δ ≤ 2 δ, andE ζ [E gc [E wc [∥e t+1 ∥ 2 ]]] ≤ (1 - δ 2 )∥e t ∥ 2 + (1 -δ)η 2 2 δ + ρκ + ρ + κ ∥∇f ( wt )∥ 2 + (1 -δ)(κ + 1)η 2 σ 2 .We simplify the notation by denotingE[∥e t+1 ∥ 2 ] = E ζ [E gc [E wc [∥e t+1 ∥ 2 ]]],and then unroll the sequence of e t back to t = 0.E[∥e t+1 ∥ 2 ] ≤ t τ =0 (1 - δ 2 ) t-τ (1 -δ)η 2 2 δ + ρκ + ρ + κ ∥∇f ( wτ )∥ 2 + (1 -δ)(κ + 1)η 2 σ 2 ≤ (1 -δ)η 2 2 δ + ρκ + ρ + κ t τ =0 (1 - δ 2 ) t-τ ∥∇f ( wτ )∥ 2 + (1 -δ)(κ + 1)η 2 σ 2 t τ =0 (1 - δ 2 ) t-τ ≤ (1 -δ)η 2 2 δ + ρκ + ρ + κ t τ =0 (1 - δ 2 ) t-τ ∥∇f ( wτ )∥ 2 + 2(1 -δ)(κ + 1)η 2 σ 2 δ . ▷ t τ =0 (1 -δ 2 ) t-τ ≤ 1 1-(1-δ 2 ) Taking η ≤ 1 10L( 2 δ +ρκ+ρ+κ) , we have E[∥e t+1 ∥ 2 ] ≤ 1 -δ 100L 2 2 δ + ρκ + ρ + κ t τ =0 (1 - δ 2 ) t-τ ∥∇f ( wτ )∥ 2 + 2(1 -δ)(κ + 1)ησ 2 δ10L 2 δ + ρκ + ρ + κ ≤ 1 -δ 100L 2 2 δ t τ =0 (1 - δ 2 ) t-τ ∥∇f ( wτ )∥ 2 + 2(1 -δ)(κ + 1)ησ 2 δ10L 2 δ ≤ (1 -δ)δ 200L 2 t τ =0 (1 - δ 2 ) t-τ ∥∇f ( wτ )∥ 2 + (1 -δ)(κ + 1)ησ 2 10L .Then, stacking E[∥e t ∥ 2 ] and taking total expectation, we haveT t=0 E[∥e t+1 ∥ 2 ] ≤ (1 -δ)δ 200L 2 T t=0 t τ =0 (1 - δ 2 ) t-τ ∥∇f ( wτ )∥ 2 + (T + 1)(1 -δ)(κ + 1)ησ 2 10L ≤ (1 -δ)δ 200L 2 T t=0 +∞ τ =0 (1 - δ 2 ) τ ∥∇f ( wt )∥ 2 + (T + 1)(1 -δ)(κ + 1)ησ 2 10L ≤ 1 -δ 100L 2 T t=0 ∥∇f ( wt )∥ 2 + (T + 1)(1 -δ)(κ + 1)ησ 2 10L .Putting all the ingredients together and taking total expectation, we haveEQUATIONFinally, using Lemma A.1, we have1 T + 1 T t=0 E[∥∇f ( wt )∥ 2 ] ≤ 8 T + 1 E[f (w 0 )] -f * + f * -E[f (w T +1 )] η + 8(11 -δ)(κ + 1)Lησ 2 20 ≤ 80L 2 δ + ρκ + ρ + κ (f (w 0 ) -f * ) T + 1 + 4σ (11 -δ)(κ + 1)L(f (w 0 ) -f * ) T + 1 .To further demonstrate the effectiveness of SDP4Bit in enhancing training efficiency, we present the relationship between wall clock time and training loss in Figure 9 . To further illustrate the impact of the Hadamard transformation on (de)quantization performance, we provide (de)quantization throughput experiment in Table 5 , which is tested on an A100 GPU. In the experimental section, we utilize a total of six different sizes of GPT models. Their model configurations are detailed in Table 7 .For the accuracy experiments, we standardize the batch size to 256, and set sequence length to 2048. We use AdamW [17] optimizer in all the experiments. The detailed training parameters are listed in Table 9 .In the throughput experiments, to more clearly study the communication bottleneck and ensure consistency across different GPU counts, we set the accumulation step to 1. The batch size is adjusted according to the number of GPUs, and the sequence length (micro batch) is uniformly set to 2048. Due to the different number of GPUs per node in the two architectures, we adjusted the tensor parallel size (TP) and pipeline parallel size (PP) accordingly, referencing [26] , to achieve the highest throughput. Specifically, the maximum tensor parallel size is 4 for the 4xA100 environment and 8 for the 8xH800 environment. See detailed parameters in Table 8 . Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: All reviewers have given favorable ratings for this paper, ranging from 5 to 7.
Average Rating: 6.00 (Min: 5, Max: 7)
The following strengths of the paper have been highlighted. However, as there are few strong recommendations for this paper, an acceptance as a poster presentation is considered appropriate.

1: This paper proposes an innovative approach to reduce communication overhead in large language model (LLM) training with Sharded Data Parallelism (ShardedDP). The proposed method introduces quantization based on weight differences and two-level smooth quantization, compressing communication to nearly 4 bits without significantly compromising accuracy.

2: The proposed method demonstrates up to a 4.08x increase in end-to-end throughput when training GPT models with up to 6.7 billion parameters. Additionally, theoretical convergence guarantees are provided, and the impact on training loss is confirmed to be minimal.

3: The proposed method involves a system and algorithm co-design that includes GPU kernel implementation and runtime optimization, minimizing computation overhead. This aspect is highly valued as it contributes not only theoretical ideas but also practical implementation and optimization.

4: The method has been evaluated at a scale of up to 128 GPUs and is applicable to models of varying sizes. By improving the efficiency of communication compression, the approach proves useful in a wide range of training scenarios.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel approach to reduce communication overhead in Sharded Data Parallelism (ShardedDP) for training large language models (LLMs). It proposes two key techniques: quantization on weight differences and two-level gradient smooth quantization, which effectively compress weights and gradients to nearly 4 bits without compromising training accuracy. The method is implemented within the Megatron-LM framework and achieves up to 4.08× speedup in end-to-end throughput on 128 GPUs. The paper also provides theoretical guarantees of convergence and demonstrates negligible impact on training loss for models with up to 6.7 billion parameters.
strength_weakness: strength_weakness: 1. The connection between the convergence analysis and the quantization schemes is not entirely clear. Specifically, it is not clear if the quantizers correspond to the biased/unbiased compressors in Algorithm 4 and if so which of them is biased and which is unbiased. It appears from lines 5 and 7 of algorithm 4 that TLq is unbiased and qWD is biased but I couldn't find any mention/proof of that, so I am not sure if that is the case or if I am missing the point entirely.

2. Most of the empirical analysis is limited to the older generation of GPT models, and it is possible that may not extend as well to modern models like Mistral series, Phi series, Orca series, Llama series etc as well as the newer generation of GPT models
questions: questions: 1. Why is 4-bit quantization needed? Is 8-bit quantization not sufficient in practice, especially since we don't expect foundation models to be trained too frequently?

2. Why is inter node communication quantized to 4 bits while intra node communication is quantized to 8 bits and not the other way around?

3. I think the Hadamard transform matrix might need to be post multiplied by a diagonal matrix with randomly chosen +1/-1 values on its diagonal for it to be effective at removing outliers and flattening the data (https://arxiv.org/abs/1011.1595). Is that what has been done here? If not, why?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposed a 4bit quantization framework for sharded data parallelism training. It proposed to quantize the weight differences between iterations as the first method to reduce the accuracy degradation. It also proposed to mixed-precision quantization for intra- and inter- node communication. Experiment results show that the proposed system can achieve 2x to 4x speedup on training GPT models with size from millions to a few billions.
strength_weakness: strength_weakness: 1. The experiment results shows training/validation loss on Pile for the proposed method and baseline (FP) and QSDP and other method. The loss is an auxiliary variable to the performance of the model. A direct comparison on real dataset will be more convincing. For example, what is the accuracy of the model training by proposed methods and ZeRO/QSDP on mmlu and hellaswag/piqa/winogrande/ARC/ARE etc reasoning tasks, and gsk8k etc math tasks? This is critical to show the accuracy/precision improvement over other 4-bit communication methods.

2. The paper shows the speedup of the proposed methods, but I  would like to know which results in Table 4 is the training throughput for ZeRO++ and QSDP (if applicable)? I would like to see a comparison for accuracy-training_speed between the proposed method and ZeRO /QSDP. Furthermore, is there any results on the training throughput (tokens/sec) at certain sequence length (instead of training TFLOPs)?

3. The paper introduces a few method, including qWD, two-level quantization, Hadamard transforms, etc. I would like to know which ones are the invention of the paper? For example, using Hadamard transform to handle the outlier has been known for a while.
questions: questions: Please refer to the session "Weakness"
limitations: limitations: n/a
paper_summary: paper_summary: Thank you for submitting your paper to Neurips 2024. The paper proposes SDP4Bit, a communication quantization method that mitigates accuracy loss by weight difference quantization and 8-bit intra-node and 4-bit internode quantization. The authors provide convergence analysis to show that SDP4Bit is compatible with both biased and unbiased compressors and experimental results to show that SDP4Bit outperforms existing solutions.
strength_weakness: strength_weakness: 1. Missing key ablation studies. The two proposed strategies in this paper are weight difference quantization and two-level int8-int4 all-to-all gradient averaging. So I suggest the authors add the following two experiments to compare the accuracies.
- Running ZeRO++ with two-level int8-int4 all-to-all gradient averaging and its original quantization method
- Running ZeRO++ with its original int4-int4 all-to-all gradient averaging and the weight difference quantization.
2. The figures are too small to read.
3. Related work could add AG-SGD, which compresses changes in activations rather than the activations themselves, so it does not rely on the assumption that gradients are unbiased.
[1] Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees
questions: questions: 1. Under what assumptions are weight differences generally easier to quantify? I've read several papers based on this intuition, but none that provide a comprehensive analysis of this intuition.
2. If you only need to assume that the magnitude of the weight difference is smaller than the weight, how do you ensure this?
3. If you ensure this via the learning rate, I think it will potentially lead to higher wall clock convergence time.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper presents a novel scheme for reducing the communication cost of training LLMs using FSDP approach. To this end, the authors suggest to quantize the weights differences (instead of the weights themselves) and applying Hadamard transformations for making the gradients smooth. They show up to ~4x training speedup on 128 GPUs.
strength_weakness: strength_weakness: 1. Although, the idea is interesting, I think the authors should provide more ablations on various aspects of their design choice (see questions section) to elaborate the approach better.
2. To motivate the problem more, I would like to see some practical analysis of the communication volume based on the number of nodes. For example, how much data (in GB) will be sent between gpus in a single node (and between different nodes) when we scale to a high number of gpus.
questions: questions: I have some questions/ideas about the paper. I would be happy if the authors answer them.

1. The paper applies symmetric quantization for all schemes. However, in the communication reduction techniques, it is natural to use asymmetric quantization (which is a bit more accurate). Also, stochastic rounding could be another option to have unbiased gradients. Have you tried those schemes to see the accuracy changes? Maybe you don't need to apply hadamard in that case and the accuracy will be recovered!

2. The main motivation behind the hadamard smoothing is to remove the outliers. This is studied in several works [1, 2, 3, 4] (which are not cited here also). My main question is why we do not see an analysis of the gradient distribution in the paper so we can make sure that the hadamard is important.

3. There isn't any study on the overhead of applying Hadamard scheme. I agree that there are fast hadamard kernels when the input is power-of-two. However, it is not clear for me if your input is not power-of-two. Also, there isn't an ablation on the size of the Hadamard in that case.

References:

1. Training Transformers with 4-bit Integers (https://arxiv.org/abs/2306.11987)
2. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (https://arxiv.org/abs/2404.00456)
3. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (https://arxiv.org/pdf/2402.04396)
4. SpinQuant: LLM quantization with learned rotations (https://arxiv.org/abs/2405.16406)
limitations: limitations: Yes. The author discuss that the work focuses only on communication reduction.

Comments: title: Response to rebuttal
Reply
comment: Thank you for adding ablation studies and the explanation of "easier to quantize". I will maintain my score.
Thank you for addressing my concerns. As I had already recommended accepting the paper, I will keep my score.
Thank you for your insightful review and valuable advice, which have been very helpful in improving our paper.
Thanks for your helpful answers and explanation. I will raise my score to 7.