HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs
Authors: Saleh Ashkboos
Mahdi Nikdan
Soroush Tabesh
Roberto L. Castro
Torsten Hoefler
Dan Alistarh
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
Quantization
Rotation
Fine-tuning
Full Fine-tuning
PEFT
Primary area: deep_learning
TL;DR: A low-precision scheme for fine-tuning LLMs

Abstract: Quantized training of Large Language Models (LLMs) remains an open challenge, as maintaining accuracy while performing all matrix multiplications in low precision has proven difficult. This is particularly the case when fine-tuning pre-trained models, which can have large weight, activation, and error (output gradient) outlier values that make lower-precision optimization difficult. To address this, we present HALO, a new quantization-aware training approach for Transformers that enables accurate and efficient low-precision training by combining 1) strategic placement of Hadamard rotations in both forward and backward passes, which mitigate outliers, 2) high-performance kernel support, and 3) FSDP integration for low-precision communication. Our approach ensures that all large matrix multiplications during the forward and backward passes are executed in lower precision. Applied to LLaMa models, HALO achieves near-full-precision-equivalent results during fine-tuning  on various tasks, while delivering up to 1.41x end-to-end speedup for full fine-tuning on RTX 4090 GPUs. HALO efficiently supports both standard and parameter-efficient fine-tuning (PEFT). Our results demonstrate the first practical approach to fully quantized LLM fine-tuning that maintains accuracy in INT8 and FP6 precision, while delivering performance benefits.

HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs
Saleh Ashkboos*1 ETH Zurich
Mahdi Nikdan" ISTAustria
Soroush Tabesh ISTAustria
Roberto L. Castro ISTAustria
Torsten Hoefter
Dan Alistarh
ETH Zurich
ISTAustria & Neural Magic
Abstract
Quantized training of Large Language Models (LLMs) remains an open challenge, maintaining accuracy while performing all matrix multiplications in low preci- sion has proven difficult. This is particularly the case when fine-tuning pre-trained models, which can have large weight, activation, and error (output gradient) out- lier values that make lower-precision optimization difficult. To address this, we present HALO, new quantization-awwe enables accurate and efficient low-precision training by combining 1) strategic placement of Hadamard rotations in both forward and backward passes, which mitigate outliers, 2) high-perfie high-performance kernel support, and 3) FSDP integration for low www-reccisinn communication. Our approach ensures that large matrix multipli- cations during the forward and backward passes are executed in lower precision. Applied to L.AAMA-famlly models, HALO achieves results during fine-tuning on various tasks, while delivering up to 1.41 x end-to-end speedup for full fine-tuning on RTX 4090 GPUs. HALO efficiently supports both standard and parameter-efficien fine-tuning (PEFT). Our results demonstrate the first practical approach to fully quantized LLM fine-tuning that maintains accuracy in INT8 and FP6 precision, while delivering performance benefits.
training approach for Transformers that
waall-precisionequueee
Introduction
The high performance of large language models (LLMs) across a wide series of tasks comes with considerable computational costs; reducing them is one of the key directions in Machine Learning Systems research [9; 16; 21]. For LLM inference, a standard acceleration approach has been the quantization of weights and activations, which reduces the precision which they are stored and potentially also computed over, e.g. [17; 42; 35:5; 3].
By contrast, much less is known about quantization in the context of LLM training. While the recent DeepSeek breakthrough [11] has showed that 8-bit foottin-poiit (FP8) pre- pre-training of large models can be done both accurately and efficiently. it is not known how to extend this result using the more widely-supported 8-bit integer (INT8) format, or lower peerr-preiison 6-bit formats such as MXFP6 [38; 41]. Intuitively, accurate and efficient quantized training is much more challenging than quantized inference: while in inference a single large matrix multiplication occurs in lower precision, for training, all three matrix multiplications (one during the forward pass and two during back-propagation) must be executed in lower precision. This creates critical issues both in terms of accuracy- quantizing weights, activations, and errors induce significant training instability-
"Equal contribution
Conresonneence to: salehashibboss @ inf ethz.ch
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Errors (Kurtosis: 118.4)
Reft-hand-side Hadamard (Kurtosis: 76.48)
Left-hand-side Hadamard (Kurtosis: 3.59)
00
35
3.5
5.
3.0
25
27
20
Channel iesolute
15
2.0
1.0
65
A4
0.
00
60
6/
511
511
Channel
301
Channel
352
183
38
Token
Token
Token
127
327
127
Figure 1: Largest magnitudes of first 512 channels in output gradients, errors, of the mlp output (hown_proj) in the 15-th layer of LLAMA3-88 8B model over 128 tokens in the 60th step of the ViGGO fine tuning (see Appendix A.4 for other data types). The outliers are propagated across columns and can be mitigated after applying left-hand-side Hadamard transformations.
but also in terms of performance-as the overheads of switching between representations can negate the performance gains of lower-precision computation.
This accuracy problem is especially challenging in the popular practical scenario where the user fine-tunes a pretrained model: as we illustrate this paper, since pretrained LLMs often have large outlier values in the weight, activation, and eπror distributions [37; 42; 34; 28], stable results much harder to achieve compared to from-scratch quantized pre-training.
Contributions. We present a new quantized training technique called Hadamard-Assiser Low- precision Optimization (HALO), which modifies the structure of Transformer- in transparent fashion, allowing them to be fine-tuned in lower precision (INT8, or even FP6), with minimal accuracy loss. Importantly, we so while performing all large matrix multiplications in lower precision. HALO applies equally well to full and parameter-efficien fine-tuning.
models [36]
HALO starts from an in-depth analysis of the quantization error sensitivity of different internal representations (weight, input, and output gradients) during back buck-propagation [33; 22). We identify the forward pass and output gradients key sources of sensitivity to quantization, each exhibiting distinct outlier patterns. Specifically, while applying Hadamard transformation is sufficient to mitigate outliers in the forward pass, addressing outliers in the output gradients during the backward pass requires applying the Hadamard matrix from the left side, referred as Left-hand-side Hadamard (Figure 1). This results two "levels" HALO, which can be chosen based on precision, data format, and acceptable accuracy drop. We complement our algorithmic contributions with efficient kernel support, allowing for computational speedups. addition, we integrate HALO with the Fully Sharded Data Parallel (FSDP) scheme, enable further savings via low-precision communication.
In summary, our contributions are as follows:
We consider the challenging problem of quantized fine-tuning of a pre-trained model using only lower-precision multiplications, and analyze the impact of quantization errors model accuracy during training, specifically linking them to the presence of outliers across various model dimensions and internal states. Starting from this analysis, we propose using left- and right- hand-side Hadamard transforms to address outliers over matrix rows and columns, respectively. We show how these transforms be inserted "strategically" in the Transformer architecture for both forward and backward passes, minimizing overheads.
2. We then explore different outlier protection levels for HALO, based on the number and placement of Hadamard transforms during training: Intuitively, HALO-1 strikes a trade-off between accuracy and performance that is well-suited for distributions with moderate dynamic ranges (such as FP6). contrast, HALO-2 employs rotations to protect all multiplications during both the forward and backward passes, making it more suitable for integer representations (such as INT8). Importantly, the levels are independent of the precision used, allowing us to adjust them to recover final accuracy while maximizing end-to-end speedup. HALO is compatible with full fine-tuning (FFT) and PEFT methods, and is backed by new efficient GPU kernel implementations, currently aimed at NVIDIA RTX GPUs. Moreover, the fact that all computation happens in quantized form offers

---

the opportunity to also perform quantized communication during sharded (FSDP) training, for which we also add support HALO.
3. We examine the accuracy and performance of HALO for fine-tuning LLAMA and Qwen-family models [13; 43], via both FFT and PEFT. We observe that HALO closely tracks the accuracy of full-precision variants across a wide series of tasks, improving upon the best known prior methods [38; 41] on the more challenging INT8 and FP6 formats. We provide performance measurements per module and end-to-end, with peak speedups of 1.82x and 1.41x x for INT8, relative to well-optimized half-precision baseline.
Overall, our results show for the first time that is possible to perform fast and accurate fine-tuning while the majority of the forward-backe even if the model initially has an outlier structure in weights, activations, and errors.
computation (all linear modules) is in lower precision,
2 Background
Related Work. Inference quantization methods aim to compress either the model weights [17: 14; 35] or jointly quantizing weights and activations, allowing for lower-precision forward computation [42]. It is known that activation quantization is challenging due to "outlier features" [37] of much larger magnitude. Recent works apply transforms to mitigate such features on top of pre-trained models. Chee et al. [5] observed that the magnitude of the largest elements (outliers, roughly defined as values 2 10 larger than the value average in the considered tensor) in the weight matrices W that we wish to quantize for inference can be reduced by applying orthogonal transformations to W from both sides, a method known as Incoherence Processing [6]. QuaRot [3] mitigates the impact of outliers in both weights and activations by applying (randomized) Hadamard rotations that are fused into the weights- a technique known as computational invariance. This approach enables most inference computations to be performed in 4 bits without incurring the transform overhead during inference. SpinQuant [23] builds on a similar idea but trains a subset of orthogonal transformations. Unfortunately, the QuaRot approach cannot be used for low-precision truining (see Section 3.1).
Performing low-precision computation during the backward pass is strictly harder than quantized inference, for instance due to the very high dynamic range of gradients with respect to layer out- puts [24; 7]. LM-FPS [32] trains large-scale models from scratch using FP8 on 40-100B of tokens, while Fishman et al. [15] demonstrates that FP8 training becomes unstable when training with over 250B tokens, and proposes a new activation function to address this issue. As such, their technique cannot be used to fine-tune an already-trained trained model.
Integer (INT) quantization is a promising direction due to broad hardware support [4], at the cost of narrower dynamic range. SwitchBack [38] trains vision models with up to 1B parameters from scratch, but only quantizes two out of three matrix multiplications in the linear modules in INT8, while retaining the third in high precision. Jetfire [41] proposes a more complex 2D block-wise quantization approach for training from scratch in INT8. Jetfire obtains good accuracy and significant end-to-end speedups (1.4 1.5x) ); yet, their method was not tested for fine-tuning, and changing the entire data flow to INT8 comes with significant challenges. The recent DeepSeek FP8 training [11] adopts similar approach to Jetfire, using blocks of size 256 and a different implementation.
We compare SwitchBack and JetFire/DeepSeck baselines, in the context of fine-tuning HALO achieves similar or higher accuracy than SwitchBack, while consistently outperforming it in terms of runtime, due executing all multiplications in low precision. Relative to JetFire/DeepSeek, we validate that FP8 training is indeed lossless, but that both methods are lossy when applied to INT8 or FP6 formats. For INT8/FP6, HALO offers higher accuracy.
Generally, efficient line-tuning is an important workload in the context of high-quality open-weight models. Most recent work focuses on making fine-tuning more efficient by reducing memory usage [12; 27] through PEFT-style schemes [19]. HALO is completely compatible with PEFT schemes. We present results from integrations with LoRA and QLoRA, but our focus is on accelerating the actual computations during fine-tuning, which remains a less-explored area.
Linear Layers. Let matrices W E RR X E Rbxm and Y E R be the weights. inputs, and outputs of an n x m linear layer acting on an input with batch size The forward and backward passes include the following matrix multiplications, in Py Torch notation [31]]
3

---

Y= X WT
(1a)
G - G
X
(1b)
Ex= W (Ic)
where G, Ex and Ey are the gradients w.r.t. the weights, inputs, and outputs (where the last two are known as errors), respectively. The first matrix multiplication occurs during the forward pass, denoted by F, while the last two occur during the backward pass: one for computing the weight gradients, denoted by G, and the other for computing the errors, denoted by E.
Hadamard Transformations. For a fixed dimension d, normalized Hadamard matrix Ha is an orthonormal matrix where H = I. When d = 2", Ha is called the Walsh-Hadamand Hadamard matrix; its and can be built via the recursive construction Ha = H2 @ Hd/2-
entries are + vd
Assume matrix A <Rdx Then, right-hand-side Hadamard transformation of A applies the linear transformation h(A) = AH When d is power-of-two, Ha is the and the above transformation can be done using recursive algorithm with Od d) operations [45]. Following [3], when d is not a power of two, we factorize it as d = m,, where m is the size of known Hadamard matrix. We then apply the Kronecker construction: Ha H2n @ H We also use the left-hand-side Hadamard transformation, defined as h' (') H A, which can be calculated by transposing h(A?)
matrix,
Fine-TTuning. Full fine-tuning (FFT) involves adjusting all the parameters of a pre-trained model on downstream task, but can be infeasible for LLMs on consumer hardware. Parameter-efficient fine-tuning (PEFT) methods such as LoRA [19] focus reducing memory usage during fine-tuning. A LoRA linear layer is parameterized by a non-traine non-trainable weight matrix W E R and trainable components U grxm and V E |@ with << min(m, n). In the forward pass, the input tensor X will be passed through the frozen and low-rank weights and the output will be calculated Y = X WT + (X U²)- VT. In the backward pass, the gradient will calculated only for the low-rank matrices. For the details of the operations, see Appendix A.3.
Full and Parameter-Efiiche
3 Method
In this section, we introduce our proposed HALO method for low-precision fine-tuning of pre-trained models. We begin by analyzing the key challenges of applying quantization during fine-tuning. Next, we present our solution to these challenges and define the levels of HALOO Finally, we discuss the details of the implementation and integration quantized communication.
3.1 Design Space and Challenges of Jow-Precision Fine-Tumee Tunn-Tuinng
Our primary goal is to apply Hadamard transformations reduce quantization error while performing the forward and backward passes of linear layers in low precision. This approach introduces several challenges that should be addressed to achieve an optimal balance between accuracy and performance. In this (and the next) section, we outline these challenges and propose solutions for each.
Challenge 1: Orthogonal Transformation Absorption. QuaRot [3] eliminates the overhead
of applying Hadamard transformations during inference by absorbing them into the weights of preceding layers. This is feasible when the network exclusively uses RMSNorm as its normalization module which has a distributive property: for any matrix X and orthogonal matrix Q, we have RMSNorm (XQ)= RMSNorm(XX (.)Q. As result, if Q is absorbed into the output dimension of weight matrix in the previous layers, it will be preserved across subsequent for explicit transformations during inference. This technique is known computational invariance [2]. However, this property does not hold during the backward pass, as the derivative of RMSNorm lacks the same distributive behavior (see Appendix A. 1). Therefore, QuaRot cannot be directly extended to the backward pass to eliminate the cost of applying Hadamard transformations.
removing the need
Challenge 2: Hadamard Combinations and Overheads. For given matrices A E Rmmk and
B E kk our goal is to perform matrix multiplication Y AqBo in low precision, where Aq and BQ the quantized versions of A and B, respectively. In the No Hadamard Case (denoted Y), we directly apply the quantization function and compute Aq without any Hadamard transformation. In the Left Case (denoted H HY), apply left-hand-side Hadamard transformation

---

Woigh Froftan Cheite Suritaccty
Woight Grostten Creinn Strileetty
Craalenn Stmilactty
AV9
2.6
244
Nrw: 9:802
02-
10 20
20
90
16
20 90
10 35 25 80
Layer
Lager
Tea
181
Figure 2: Cosine similarity between the weight gradients of the baseline model (BF16) and the quantized model when quantization is applied during the (a): backward pass, (b): forward pass, and (c): forward pass with Hadamard transformation in single fine-tuning step of LLAMA3-$Bon the GSM8K dataset. Compared to the backward pass, the forward pass is more sensitive to quantization. We improve the results by applying Hadamard transformation during the forward pass (#). For each case, present the weighted average (over # parameters) for all linear modules in each layer.
of size to A before quantization and compute H ( A qq In the Right Case (denoted Y"), we apply right-hand- hand-sideee Hadamard transformation of size to B, perform Aq (BHH))) (BHH)QF Q
Lastly, in the Middle Case (denoted Y). we apply Hadamard transformations of size k from the right A and from the left B. computing (A ( yields 512 possible modes, including the "No- Ho-Haaamardd case. Selecting the optimal insertion strategy challenging because: (1) we to minimize the number of Hadamard transformations to reduce computational overhead; and (2) we must carefully place them to suppress outliers and maintain stability during low- lowpprecision training (see Appendix A.7 for an ablation study over different combinations). Naively evaluating all combinations of Hadamard placements for each fine- tuning task is clearly impractical.
Any combination of these placements
Challenge 3: Communication and Memory Issues. Fine-tuning large models typically requires multiple GPUs, which must communicate over a network to perform operations like weight gathering (AllGather) and gradient aggregation (AllReduce). These communication steps introduce significant overhead during training. Additionally, storing activations for the backward pass (especially with large batch sizes) can become a major memory bottleneck. Therefore, HALO has to also consider communication and memory efficiency constraints in the method design.
3.2 HALO Solutions for Fine-tuning
As discussed in the previous section, quantized fine-tuning presents several challenges. In this section, we present the main steps of designing HALO to address the above challenges during fine-tuning.
Step 1: Sensitivity Analysis of Forward vs. Backward Quantization.
We begin by identifying the most quantization and backward passes. While is known that activations (X) and emor gradients (Ey) exhibit outliers in their columns and rows, respectively [3; 39; 7], it remains unclear which specific matrix multiplications are most vulnerable to quantization effects. To address this, we study the cosine similarity between the weight gradients the baseline model (no quantization) and the quantized model when quantization is applied during the forward pass or the backward pass.
matrix multiplications during the forward
Figure 2 shows that the computed weight gradients have significantly lower cosine similarity compared to the baseline model when quantization is applied during the forward pass. Specifically, when we only quantize Equations (lb-lc) in the backward pass, the average cosine similarity is 0.909. In contrast, when quantization is applied to the forward pass in Equation (la), while leaving the backward pass unchanged, the average cosine similarity drops 0.323. This drop may result from larger quantization errors in the activations, potentially caused by outliers, or from errors introduced during the network loss computation after the forward pass is quantized.
'We apply another left fetthand-sidd Hadamard transformation on the output of the low-precision matrix multiplication (approximately) cancel the effect of this transformation.
5

---

HALO
HQ-FSDP (Forward Pass)
(Input Gradient Calculation)
HALD
Mubie Aitiiiiation
WH
Wi
H
Aneidax
Quantzs
( (Wo
(WH)a
By
Max Recuce
Quartzood All Gather
0-2
Matri
WH H By
W2
Local
P.8
Quantzs
(WoD
(WH)a
Kusspiaation
H
highthnndeete Hadamard
Lett handaaddssce Hadanard
Figure 3: Left: Hadamard transformations in HALO levels during the input gradient calculation. Hadamard transformations already applied to the weights during the forward pass. Right: Forward pass in HQ-FSDP for two GPUs: Each GPU performs a right-hand side Hadamard transformation and computes the absolute maximum (AbsMax) 8; over its local weight shard W,. Then, all GPUs participate in an All-Reduce operation and compute the global maximum absolute value 8. Each GPU uses this global value to quantize its weight shard. Finally, an All-Cather operation is performed on the quantized shards (W,H)a All GPUs use the same scales to quantize the weights the backward pass after applying Hadamard transformation.
We conclude that mitigating outliers is key during the forward pass, and recover most of the averaged cosine similarity (0.883) by applying Hadamard transformations during the forward pass on the weights and inputs.
Step 2: Memory- and Communication Effficien Backward Pass.
In Step 1, we apply hight-hand-side Hadamard transformation to both weights and activations. Due to the orthogonality of the Hadamard matrix, these transformations cancel out in Equation (la). In the next step. we retain the quantized transformed weights ((H))q) and activations ((H))) for the backward pass and perform the computations defined in Equations (Ib-1c) using these tensors. This ensures consistency between the forward and backward passes, as both operate on the same (quantized) inputs and weights, while also enabling efficient reduction in storage and communication.
To recover the true weight and input gradients in Equations (1b-lc), an additional right-hand-side Hadamard transformation must be applied to the outputs of the corresponding matrix multiplications in the backward pass, introducing two extra Hadamard transformations into the overall scheme.
Step 3: Outlier Mitigation for Output Gradients (Errors) in the Backward.
With the first two steps, we can stabilize fine-tuning at precisions with moderate dynamic ranges (such as FP6). However, for narrower dynamic ranges (e.g., INT8), it becomes necessary to mitigate outliers in the error gradients during the backward pass.
Figure e-Left shows that, unlike activation tensors X, in the error tensors Ex the outliers are propagated across columns (channel/feature dimension). Then, using right-hand-side Hadamard transformation does not solve the outlier issue as such transformation rotates the input rows (Figure I-Middle). Thus, we apply left-hand-side Hadamard transformation, h' (Ey), which mitigates the етror outliers by rotating the column vectors (Figure I-Right).
We avoid applying the Hadamard transformation during weight gradient computation because (1) computing input gradients is more sensitive to quantization, as these gradients are propagated to previous layers, and (2) performing seft-hand-side Hadamard transformation is more computationally expensive due to the need for two transpositions.
3.3 The HALO Method
Full Fine Tuning, Based on the above steps, we define two levels within HALO, corresponding to data formats with moderate and narrow dynamic ranges, respectively as follows:
At the first level, denoted by HALO-I(H, E", G.), we employ the middle case during the forward pass to mitigate issues with weight gradients (Figure 2c). Specifically, right-hand- side Hadamard transforms are applied both input and weight matrices prior to quantization.
This introduces two implementation concerns: 1) Low-Precision Communication: We apply

---

a right-hand-side Hadamard transform to the weights in Equation (1c) to facilitate integration with HQ-FSDP and Activation Checkpointing 2) Memory Reduction: To improve memory efficiency, we apply a right-hand-side Hadamard transform to the input during the backward pass in Equation (1b). This allows us to reuse the quantized inputs from the forward pass, eliminating the need to keep the high-precision inputs in memory.
2. Finally, mitigate outliers in errors, we define the highest level by applying left-hand-side Hadamard transformation to E, denoted by HALO0-(FF, H c.).
The main difference in the HALO levels applying Hadamard transformations during the error calculations. Table 3 (in Appendix A.2) summarizes the key modifications introduced to the matrix multiplication computations across different HALO levels. The primary distinction between HALO-1 and HALO-2 is the application of a sidt-hand-side Hadamard transformation to the errors during input gradient calculation, as shown in Figure 3-Left.
Parameter Efficient Fine Tuning. To accelerate parameter-efficient fine-tuning (PEFT), we apply (F. Hg c) on the large matrices while maintaining low-rank operations in high precision due to their inherent efficiency. For the weight quantization, we apply right-hand-sin Hadamard transformation and quantize the frozen weights once and store the quantized weights before fine-tuning. We then apply single right-hand-side Hadamard transformation during the forward pass on the inputs and two Hadamard transformations during the backward pass on the errors. Finally, we keep the gradient calculation in high precision as it uses only low-rank matrix multiplication. We denote this variant scheme by HALOPEFT- We present the details of the Equations in Appendix A.3.
3.4 HQ-FSDP: HALO Quantized Communication and Memory Reduction
In this section, we describe how HALO leverages quantized Fully Sharded Data Parallel (FSDP) to reduce communication overhead, and quantized activation storage minimize memory usage.
FSDP Integration. Fully Sharded Data Parallel (FSDP) [48] is a common distributed training strategy for LLM fine-tuning. In FSDP, model weights are sharded (i.e., distributed) across multiple GPUs. Whenever necessary, a subset of the weights (typically corresponding to a transformer block) is all all-gathered, enabling every GPU to have the full weight subset required for an operation. This communication occurs before the forward and backward passes of each block. After the backward pass, gradients are reduce-scattered ensuring that each process maintains the averaged gradient for its own shard (Figure 3-Right).
Since HALO requires only the quantized weights for performing the F and E operations, it allows for low precision all-gather communications. We refer to this approach as Hadamard- -uantaiiee FSDP (HQ-FSDP) and implement it as follows: (a) if necessary, each process applies right-hand Hadamard transformation to its shard (depending on the HALO level implemented), (b) each process computes local guantization scale for its shard, (c) the scales are max-reduced to ensure that every process has the global scale per weight matrix, (d) each process quantizes its shard with the global scales, and (c) the oowprecisioon quantized weights are communicated. This approach significantly reduces communication overhead while distributing the quantization and, possibly, the Hadamard transformation overhead across processes. Notably, the global scales calculated during the forward pass are reused in the backward pass, skipping steps (b) and (c) above. See Appendix A.5 for details.
FSDP is often combined with Activation Checkpointing (AC), which saves only selected activations (checkpoints, e.g., before and after transformer blocks) during the forward pass, reducing memory usage. Before the backward pass of each block, intermediate activations are recomputed via a second forward pass. With AC. FSDP communicates each weight only once during the backward pass, reusing it for both the second forward pass and the backward pass. Thus, when using HQ-FSDP with AC, applying the same Hadamard transformation to the weights in both F and E operations is essential to ensure communication speedup.
Activation Quantization For Memory Reduction. Xi et al. [40] have shown that up 40% of memory is used to store activations during training in LLAMA-style models. This is because the activation tensor in Equation (1a) must be stored and reused in Equation ( (1b) during the backward pass. We always store the quantized version of X for the backward pass to address this issue. When we apply right-hand Hadamard operation on X during the forward pass, we need to apply another right-hand Hadamard operation the output of Equation (Ib) to ensure identical computation.
7

---

4 Experimental Validation
We implement HALO in PyTorch [31] based on the the 11m-foundry codebase [25] for FFT, and the standard HuggingFace PEFT library for HALOPEFT- We use tensor-wise symmetric quantization (a single scale for the entire tensor) for all data types, and across all our experiments. We implement our own low-precision matrix multiplications using the CUTLASS library [29] for all linear modules (except for the LM head and embeddings) and keep the rest of the model in the original precision (BF16) during fine-tuning. For outlier mitigation, we adapt efficient Hadamard CUDA kernels [10]. We use E4M3 for the 8-bit floating- format; after comparing accuracies, we chose the E3M2 format for FP6 as the more accurate variant.
(RTN) quantization
Model, Tasks, and hyper-parameters For FFT, we evaluate HALO on LLAMA3 8B [13] as well as large scale Qwen (14B and 32B) models [44] (in Appendix 1.13), following published fine-tuning recipes [26; 27] for both FFT and PEFT. For both FFT and PEFT, we consider three standard datasets: 1) ViGGO [20]. with 5.1k training and 1.08k test samples, 2) Grade-School Math (GSM8k) [8], with 7.74k training and 1.32k test samples, and 3) SQL generation [49; 46], with 30k training and Ik test samples. These datasets are particularly interesting because they are either highly-specialized (like SQL. and ViGGO), or the pre-trained model has low few-shot accuracy (like GSM8K), making fine-tuning necessary. We use the same hyperparameters as 6F16 fine-tuning, detailed in Appendix A.6. Each experiment is repeated with 5 seeds; we report the mean and standard error.
Baselines. For full fine-tuning, we compare HALO against three main baselines: BF16; SvitchBack [38], and Jetfire [41], as described in Section 2. Switch uses row-wise quanti- zation for the inputs and errors during the forward and backward passes and performs weight gradient calculation in 16 bits. Jetfire applies 2D block-wise quantization with a block size of 32 x 32 on the linear layers and use symmetric AbsMax for quantization. Notably, for fair comparison, we maintain this scheme's 's data flow in 16 bits and do not quantize the activation functions in our Jetfire baseline, this aspect is orthogonal to scheme. Finally, compare HALO with the standard PEFT methods LoRA [19] with rank r = 8.
4.1 Low-Precision Full Fine-Tuning
Integer (INT8) Quantization. The first row in Table 1 summarizes HALO results across three tasks, where weights, activations, and errors are quantized using INT8 quantization. We uti- lize the second level of our scheme, denoted by HALO-2(F. "E". G for integer quantization and compare our results against BF16 (no quan- tization), Swi Scicck,ack, and Jetfire. Across all tasks, HALO and Jetfire achieve relative accuracy within 1% of the BF16 model. How- ever, HALO employs tensor-wise quantization for all tensors and has lower variance across different seeds. Additionally, our version of Jetfire retains the data flow in BF16, making it more accurate than the original implemen- tation. Finally, SwitchBack fails to recover accuracy on both the ViGGO and GSM8K datasets, despite using row-wise and column- wise quantization for inputs and weights, along with
Format
Method
GSMBK
VIGGO
BF16
Baneline
69.3 + 0.5
94.0 + 0.3
79.9 + 0.5
SwitchBack
6.7
61.0 28.9
79.6 : 0.8
Jetfire
08.2++0.6
93.6
80.2 0.6
HALO-2
0.13:20.1
93.8
60.1 + 0.3
SuitchBack
62.7
93.4
80.1 + 0.5
FP6
Jetfire
62.8 1.1
92.9
79.7 + 0.5
HALC-1
0.2
0.4
60.2 0.3
SuitchBack
69.1
0.2
80.4 + 0.3
FP8
Jetfire
0.4
93.1
60.2 + 0.3
No-HALO
6.2
93.2 0.2
80.3 + 0.2
Table 1:
Single-epoch accuracy results of fine- tuning with 8-bit (INT8/FP8) and 6-bit (FP6) quan- tizations. We use HALO-2(E. H Hg" G*) for in- teger quantization and HALO-1 E", c#) for FP6 (E3M2), while we use no Hadamard for FP8 (E4M3), denoted by No-HALO.
weight gradient calculations. The SQL dataset appears to be "easier", as all schemes match the baseline after epoch of FFT.
Floating-Point Quantization. Next, we study the use of floating-point representation during fine- tuning by applying FP6 (F3M2) and FP8 (F4M3) quantization. For FP6, we utilize the first level, denoted by HALO-1 H E" c where we apply the Hadamard transform on the forward pass and use the Hadamard-transte transroordd weights and inputs during the backward calculation. On GSM8K, both SwitchBack and Jetf frr have ~6.5% accuracy drop, while HALO-1 shows a 2.8% accuracy loss compared to the BF16 model. On ViGGO, HALO experiences a 0.4% accuracy drop, whereas
8

---

1.8
1.6
1.6
Speedup Rabo
1
Speedup Ratio
1.4
1.2
1.2
1.0
1,00K
LO8K
1,00
1.00
0
0.8
Switchlack
0.8
HAL
jeline
Sieedl HCC-SSPP
0.6
HALO-2
0.6-
HALL 2 HSQ
2 SSDP
BS=4
BS=8
BS=16
BS=32
BS=4
DS=8
BS=16
BS=32
(a) Linear layer (of size 4096
4096).
(b) Three consecutive transformer blocks (with/without HQ-FSDP).
Figure 4: INT8 forward + backward speedups (over BF16) for 512 sequence length in LLAMA3-8B across batch sizes (BS). HALO-2 refers to HALO-2( "e".."))
SwitchBack and Jetfire show drops of 0.6% and 1.1%, respectively. Similar to integer quanti- zation, all methods achieve accuracy within 1% of the baseline the SQL dataset. Although the peak performance of FP6 is the same as FP8 [30], FP6 provides higher memory and communication reduction with quantized activations and HQ-FSDP (see Section 3). We present the accuracy results for HALOpErrin Appendix A.3.
4.2 Speedup Analysis
We now evaluate the runtime improvements achieved with HALO. Speedups are measured on RTX 4090 GPUs with locked clocks, to reduce variance, for: a single linear layer and for end-to-end training. We compare different precisions and HALO levels against the BF16 base baselines. Unless mentioned, we fix the sequence length at 512 and control the input size using the batch size. The layer-wise and block-wise speedups averaged over 100 (+20 warm-up) and 30 (+10 warm-up) runs, respectively.
and other
Linear Layer. First, we consider single linear layer of size 4096 4096, matching the majority of linear modules in LLAMA3-8B model. evaluate our most accurate INT8 quantization scheme, HALO-2(F "E", c relative to the SOTA Jetfire and SwitchBack methods. Since Jetfire uses an INT8 data flow, we consider a setup where its inputs, outputs, and errors are in INT8, bypassing its quantization overhead: this is an "idealized" version of Jetfire to ensure fair comparison. Figure 4-(a) illustrates the speedups relative to the baseline BF16 implementation. HALO-2 attains additional 20-40% speedups compared to SwitchBack, primarily due to the higher precision matrix multiplication for weight gradient calculations SwitchBack. Additionally, for batch sizes greater than 4, HALO-2 has at least 20% more speedup than Jetfire, as Jetfire's complex 2D block-wiss wise quantization introduces significant overhcad during both quantization and dequantization. However, for a batch size of 4, Jetfire is slightly faster than HALO-2 due to the overhead from Hadamard transformations and transpositions. Switching HALO to a low- oow-precision data flow would increase speedups; leave this for future work.
The Effect of HQ-FSDP. evaluate the impact of HQ-FSDP across different bateh-sizes, we run forward and backward passes over three consecutive transformer blocks that fit in GPU memory BS=32 BS=32-using four GPUs with FSDP enabled. Figure 4-(b) reports speedups with and without quantized communication. HQ-FSDP achieves 1 1.37 to 1.43x speedups, delivering 10-40% higher speedups compared to 16-bit communication. The improvement more significant at smaller batch sizes (e.g. BS=4), communication becomes the primary bottleneck for training.
End-to-End Speedups. Finally, in Table 2, we compare the end-to-end fine-tuning speedups for LLAMA3-8B when we use INT8 HALO-2 and FP8 without Hadamard transformation, denoted by No- HALO, both which are near-lossless for the cor- responding precisions. Using four GPUs, we can fit only four samples into GPU memory, whereas with eight GPUs, can fit up to eight samples, each with
WVIDIA (RTX-4090)
BS=4
BS=8
BS=4
BS=8
INT8 (HALD-2)
1.35x
0OM
1.41x
1.41x
FP8 (No-HALO)
1.3xx
0OM
1.1x
1.39x
Table 2: End-to-end speedups one epoch LLAMA3-SB full fine-tuning with best per- forming HALO level using INT8 and FP8.

---

512 tokens. In the former case, No-HALO and HALO-2 achieve speedups of 1.35x and 1.36x, respectively. However, the speedups are higher with eight GPUs, reaching up to 1.41x, mainly due to increased communication overhead in this configuration. All experiments were conducted in a realistic setting for both four and eight GPUs, with HQ-FSDP and Activation Checkpointing enabled. W In addition, in Appendix A.9, we provide inference speed results for the model which fine-tuned using HALO, showing that the Hadamard transformations have minimal impact on inference performance.
5 Conclusion
We introduced HALO, LLM fine-tuning scheme that performs all matrix multiplications in lower- precision, leveraging Hadamard transforms to mitigate outliers. HALO uses simple tensor-wise quantization for all weights, inputs, and errors, utilizes low-precision communication (HQ-FSDP), and reduces memory usage by storing quantized activations. For INT8, HALO achieves up to 1.36x end-to-end speedup which comparable to the 1.41x speedup of FPS without any Hadamard transformations. This holds during fine-tuning of LLAMA3-8B on four and eight commodity GPUs, while maintaining accuracy close to the baseline. future work, we plan to investigate HALO for pre-training accurate models from scratch, and extend it for additional GPU hardware types.
Acknowledgements
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 program (grant agreement PSAP, No. 101002047. This research also obtained funding from the "Urban'winn: An urban digital twin for climate action: Assessing policies and solutions for energy, water and infrastructure" project, funded by the ETH-Domain Joint Initiative program in the Strategic Area Energy, Climate and Sustainable Environment.
References
[1] Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language models. arXiv preprint urXiv:2310.09259 2023.
[2] Saleh Ashkboos. Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:240l. 2024.
[3] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated Ilms. arXiv preprint arXhv:2404.00455. 2024.
[4] Mart van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel, Joseph Soriaga, et al. Fp8 versus int8 for efficient deep learning inference. arXiv preprini arXiv:2303. 17951, 2023.
[5] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
[6] Yudong Chen, Ashkan Jalali, Sujay Sanghavi, and Huan Xu. Incoherence-optiiaal matrix completion. arXiv preprint arXiv:1310.0154, 2013. Available at https: org/abs) 1310.0154.
[7] Kamran Chitsaz, Quentin Fournier, Gonçalo Mordido, and Sarath Chandar, Exploring quantiza- tion for efficient pre-training of transformer language models. arXiv preprint arXiv:2407.11722. 2024.
[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 14168, 2021.
10

---

[9] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv preprint arXiv:2307.0869/ 2023.
[10] Tri Dao. Fast hadamard transform in cuda, 2023. URL htt.ttpps///ihhhb..co///ooaaa// transform.
[11] DeepSeek-AI. Deepseek-v3 technical report, 2024. URL. 19437.
[12] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
[13] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[14] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06116 2024.
[15] Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling fp8 training to trillion-token 1lms. arXiv preprint arXiv:2499.1251,, 2024.
[16] Flashinfer.ai. Flashinfer: Kernel library for Ilm serving, 2023. URL https://tttthhtii com/
mashhinfer-ai/flahhinee
[17] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17322 2022.
[18] Jordan Hoffmann. Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
[19] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint N.Niv:2106.066880 2021.
[20] Juraj Juraska, Kevin Bowden, and Marilyn Walker. ViGGO: A video game corpus for data-to-text generation in open-domain conversation. In Proceedings of the 12th Interna- tional Conference on Natural Language Generation, pages 164-172, Tokyo, Japan, October- November 2019, Association for Computational Linguistics. doi: 10.1860.1551011 11653/v1/119966600 URL tttttp://aalanntholoo
org/ WwWWW966622
[21] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 16626,6,2000 2023.
[22] Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.
[23] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant- Ilm quantization with learned rotations. arXiv preprint arXiv:2405. 16406, 2024.
[24] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training. arXiv preprint arXiv: 171.03770 2017.
2017.
[25] MosaicML, Llm training code for databricks foundation models, 2023. URLhttps://githubb com//nosaicmm//Im-foun
[26] Artur Niederfahreehoost, Kourosh Hakhamaneshi, and Rehaan Ahmad. Fine-tuning LLMss LoRA or Full-Parameter? an in-depth analysis with Llama 2. https:// www.canyyccae.comm com/ bl blog/author=reaat aaaaaarr-rrrraaaaaaaaaaa.. ahmad, 2023,

---

[27] Mahdi Nikdan, Soroush Tabesh, and Dan Alistarh. Rosa: Accurate parameter-effiiint fine- tuning via robust adaptation. arXiv preprint arXiv:2401.0467, 2024.
[28]
Aniruddha Nrusimha, Mayank Mishra, Naigang Wang, Dan Alistarh, Rameswar Panda, and Yoon Kim. Mitigating the impact of outlier channels for language model quantization with activation regularization, 2024. URL https //axxiv./.iii////aa org/ ab//a////// /2004. 03605.
[29] NVIDIA. Cutlass: Cuda templates for linear algebra subroutines, 2017. URL https://
github. wwt.///w./////////////.//. cutlass..
[30] NVIDIA. Nvidia blackwell architecture technical brief. "tttps://resouucces.miii com/ 2024.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[32] Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang. Bolin Ni, Jingcheng Hu, et al. Fp8-Im: Training fp8 large language models. arXiv preprint ar3iv: 13..1833.. 2023.
[33] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propag errors. nature, (23(688)) 1986.
[34] Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402. /7762, 2024.
[35] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipit: Even better Iim quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04533 04336, 2024.
[36] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[37] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402-17414, 2022.
[38] Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models. Advances in Neural Information Processing Systems, 16:10211-10298 2023.
[39] Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit integers. Advances in Neural Information Processing Systems, 36:49146-49168, 2023.
[40] Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, and Song Han. Coat: Compressing optimizer states and activation for memory-efficient fp8 training. arXiv preprint arXiv:2410.1 19313, 2024.
[41] Haocheng Xi, Yuxiang Chen, Kang Zhao, Kaijun Zheng, Jianfei Chen, and Jun Zhu. Jetfire: Efficient and accurate transformer pretraining with int8 data flow and plo-klock quantization. arXiv preprint arXiv:2403. 12.40.1.2222 12422, 2024.
[42] Guangxuan Xiao, Ji Lin, Mickael Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[43] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1067/ 2024. URL https::/taaatt
org/ 0aabs/240...5 01407.1060 10671.
[44] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 11X1v:2412.15115, 2024.
12

---

[45] Raytheon Yavne. An economical method for calculating the discrete fourier transform. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I, pages 115-125, 1968.
[46] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: large-scale human- label- dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv: 1809008887, 2018.
[47] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang. and Wei Lu. Tinyllama: An open-source small language model. arXiv preprinl urXiv:2401.02385 2024.
[48] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, al. Pytorch fidp:: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:234..11277, 2023.
[49] Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, (atps/779..00111 2017.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims are justified in the paper. The practical claims are shown in the "Experimental Validation" Section (Section 4).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss that in the "Conclusion" section of the paper our future work
plan.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We explained all details for experiments in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
For the paper, we provide an anonymous code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. wwtpcccppppppeeeeeeecceccccceeee http//udddss/coeee
ssi.ssiicccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes].
Justification:
All details are presented in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: We repeat all experiments with 5 different random seeds and report the mean and standard deviations for our results (See Section 4).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes].
Justification: Our accuracy results are independent of the GPU types (as we only show the final accuracy). For speedup results, we provide details about the GPU type and configurations Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / http//neripsppe/e cttppblic////ccccccccccccccccltt c/Ethi ceGuideellnn
Answer: [Yes]
Justification:
We preserve anonymity in both text and codebase and follow the CoE.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification:
No societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification:
We use open-source models and datasets in the paper and cited all of them.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode..coo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The paper does not involve crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA].
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https //tp:///ee..../eeee Seteurips.cc/ooffeeeeee Conneerennee/22255//// /LLMM for what should or should be described.
20

---

A Technical Appendices and Supplementary Material
A.1 RMSNorm Backward Pass
For every vector x E R.d we define
fx)
|2|||
V x
2i=1
Given matrix X, the RMSNorm(X) applies Equation (2) on each row of X in the for- ward pass, For a given orthogonal transformation matrix Q. we will have f(Qx) Qf(x) as ||Qx|| = V = ||=||, which known as the distributive property. Using this fact, Ashk- boos et al. [2] showed that any orthogonal transformation Q can be fused to the weights of the previous layer, and RMSNNrm will preserve it because of the distributive property above. This approach eliminates the overhead of applying Hadamard transformations during the forward pass.
Now, consider the derivative of f(x) which is used during the backward pass of fine-tuning. For given input vector Rd, will have
T. |
Ōx
|
which does not have the above distributive property. This shows that one cannot use the fusion idea [3; 23] to merge the Hadamard transformations into the previous weight matrix during the backward pass, especially as we need to apply left-hand-side Hadamard transformations on the errors (see Figure 1).
A.2 HALO Levels for FFT
We summarize all the HALO levels for fine-tuning of linear layer in Table 3. We tensor-wise symmetric quantization function Q for all data types (input, weights, and errors).
Forward
Inpot Gradient Calculation (E)
Weight Gradient Calculation (G)
Method
Calculation (F)
No-HALO
W
(Evlg We
(E4)aXa
Suitable ranges (e.g..
Outlier mitigation during the forward
HALO-I xit aa)
(WH)
(FH) H
(
Easy integration FSDP
3. Quantized activation for memory reduction. Suitable for moderate ranges (c.g.. FP6).
HALO-2/E "g"
H H (WH)
(X
1. Most accurate schome.
Suitable for narrow ranges (e.g., INT8),
Table HALO levels for full fine-tuning (FFT). We use the quantization function Q for quantizing different data types and perform the computation with low precision. AC stands for Activation Checkpointing.
A.3 LoRA-style Linear Module and HALO for PEFT
LoRA linear module includes the following operations:
Y X WT + (X UT). VT
(4)
Ex EV E (Ex V). U Ey W,
(5)
Gv ET (X -UT).
(6)
Gu = (VT ET) X.
(7)
where GU and Gv are the gradients U and V, respectively. Since low-rank operations are fast, goal is to quantize operations not involving the the low-rank matrices U and V.
21

---

As mentioned in Section 3, we integrate HALO with PEFT by quantizing the majority of the computation while retaining the low-rank computations in high precision. In this case, the Equations (4-5) will become
Y R (XH)Q (WH) + (XU") VT, Ex R EX +H (H"Ex)o (WH)qH".
(8)
(9)
We show the above scheme by HALOPEFT.
Next, we compare HALOpEFT against LoRA [19], on GSM8K, SQL, and ViGGO for FP8, FP6 and INT8 in Table 4. For INT8 and FP8, HALOpEFT is always within the standard deviation of baseline. On GSM8K, FP6 has approximately a 2% accuracy degradation, showing the challenge of recovering high-precision accuracy with only 6 bits. For ViGGO and SQL,, both INT8 and FP6 recover accuracy with at most a 0.5% average accuracy drop.
Table 4: Single-epoch accuracy comparison between LoRA [19] and HALOParr(ours) We use rank r = 16 for adapters and apply FP6 (E3M2) and INT8 for quantization.
Precision
Method
GSM8k
ViGGO
SQL
BF16
LoRA
69.4 0.8
94.1 = 0.2
80.0 = 0.4
INT8
69.0 0.5
93.4
79.9 0.4
FP6
HALOgEFF
67.3 0.6
93.6
79.9 0.5
FP8
69.4 + 1.0
94.2 = 0.6
80.0 - 0.3
A.4
Hadamard Effect
Figure 5 illustrates the effect of applying Hadamard transformations to the right and left sides of the activations (input matrices), errors (gradients with respect to the output), and weights. For the activations, applying Hadamard on the right side effectively removes outliers, whereas for the errors. a left-side Hadamard transformation is necessary to eliminate outliers, as also noted in Figure 1.
A.5 HQ-FSDP Details
Here we discuss some details about our HQ-FSDP implementation. Master Weights. In HALO, weights are maintained in BF16. Accordingly. HQ-FSDP stores the parameters in BF16 and applies quantization before communication. Forward vs. Backward. Since the weights remain unchanged between the forward and backward passes, the quantized shard could be stored during the forward pass in each process and only communicated during the backward pass. This approach eliminates the need for a second quantization and potential Hadamard transform during the backward pass. However, it introduces additional memory overhead, as the quantized weights must be stored alongside the master BF16 weights. Additionally, since the quantization and Hadamard transform on the weights are distributed across FSDP processes, their overhead is relatively small. Instead, adopt an intermediate approach: save only the global quantization scales during the forward pass and recompute the quantization and Hadamard transform during the backward pass. FSDP Wrapping Policy. Following standard practice, we wrap each transformer block with an FSDP module (FSDP communications happen before each transformer block). However, for HQ-FSDP, we skip the layer-norm modules (keeping full weights in every process) as we do not intend to quantize them. Our experiments show that this only marginally increases memory usage and does not change runtime. Distributed Hadamard Transformation. When the scheme requires a right Hadamard transformation, HQ-FSDP applies it in distributed way: process performs W,H where Wi denotes the i'th shard. However, this requires the weight matrix to be sharded by row, i.c., each row should entirely reside within single shard. As FSDP requires equally sized shards to be fast, we dynamically insert small dummy parameter tensors of carefully chosen sizes when needed. This guarantees aligned sharding without compromising performance.
A.6 Hyper-Parameters
In experiments, we tune the hyper-parameters on the base BF16 tasks, and re-use the same values for low-precision training. We always perform single-epoch experiments using the AdamW optimizer
22

---

Activation Hldamaard
Errur No Hadamard
Weight Hodammri
25000
20.0
17.5
20000
Hy
Denaty
15000
10.0
10000
7.5
2.5
5-
0.0
0
0
Hadnmand
Hadamard
Weight Hadamand
14
4.2
8000
35-
Higir tata 62752
30
6000
Deasity
25
8-
4000
6-
4-
2-
5-
-0.5
1.0
-5.0
-2.5
0.0
2.5
5.0
-4
Te-4
10-2
Activation Right Hadamard
Earor Right Hadiamard
Weight Right Hadamard
10
0.30
8-
20000
Deusiy
6
15000
10000
2-
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Vallue
1e-1
Value
16-3
Valuue
10-1
Figure 5: The effect of applying Hadamard transformations on the left and right hand sides of the the activations, errors, and weights.
with 31 = 0.9, .2 = 0.999, and a linear learning rate warm-up of 20 steps. The batch size and sequence length fixed at 32 and 512. For FFT, we choose learning rates 4 x 10-3 6 10-6, and 3 x 10-5 for ViGGO, GSM8k, and SQL, respectively, and for PEFT LoRA experiments, we choose the learning rate 6 10-4 and LoRA rank of 16 for all datasets. These learning rates were found to be the best using a grid search within the range [10-6, 10-3] of 20 uniform log- linearly separated grid points, trained and evaluated using the non-quantized BF16 training precision.
A.7 Ablation study on Hadamard schemes
Figure 6 presents various combinations of applying Hadamard transformations during the forward and backward passes. As shown in Figure 6-Left, applying Hadamard in the forward pass crucial. For the backward pass, Figure 6-Right indicates that HALO-1 EH c*) and HALO-2(# нyи ( are the optimal configurations, balancing the minimization of Hadamard transformations with system-level considerations such memory usage and communication compression.
23

---

Without Forward M Hadamard
With Porward M Hadamard
LR
LR
67. 67.9620.41
08.33:0.16
68, 28.4320.63
HALO-2
((rpput Grad) scheme
0,00=0.00
0.9220:42
1.35&9.72
67.3220.66
68.07=0.69
67.85
67. 07.71
06
HALO-4
1.33:1.19
1. 1.74 45
2.59x1 50
2.51
x
67 02.8220.224
60 67.6770.33
65 65.0..311
68t 08.07700.00m
62
3,00g:1
1.96
3.248
3.09e1
I0X 63.17:0.07
07
63 00--22:000
60
1R
(Weight: Grad) scheme
G (Weight Grad) scheme
Figure 6: Different combinations of applying Hadamard transformations during the forward and backward passes of training LLAMA3-88 on GSM8K using INT8 precision. Here, "L" denotes applying a Hadamard transformation side, and "M" refers to the middle case. "LR" represents applying Hadamard transformations on both the left and right sides, while "O" denotes the case where no Hadamard transformation is applied.
the left-hand side, "R" indicates applying it the right-hand
A.8 Ablation Study on HALO Levels
One interesting question concerns the comparison between the different levels of HALO introduced in Section 3.3, that HALO-0(F E, G), HALO-I(F, E cd), and HALO-2 "E", c²) used during the fine-tuning of FP8, FP6, and INT8, respectively.
Table 5 shows the effect of using different levels for INT8 and FP6, illustrating the natural finding that a higher HALO level results in higher final test accuracy. For INT8 precision, HALO-0 fails recover accuracy, leading to an approximate 40% drop in accuracy (on average) on our datasets. At the next level, HALO-1 recovers approximately 37% of the above accuracy gap by applying right-hand-se Hadamard transformations on the weights and inputs during both the forward and backward passes. Finally, HALO-2 applies left-hand-side Hadamard transformations on the errors, achieving within I % of the BF16 accuracy. For FP6 precision, although HALO-2 achieves higher accuracy on the GSM8K dataset, believe HALO-1 provides better trade-off between accuracy and the number of Hadamard transformations.
Table 5: The accuracy effects of using different HALO levels within each quantization precision. The selected level for each precision is presented with bold text. We exclude FP8 experiments as HALO-0 recovers the BF16 accuracy.
Precision
Method
GSMBk
ViGGO
SQL
BF16
Baseline
69.26 :: 0.51
94.02
0.29
79.83 0.49
HALO-0
62.32 +: 0.65
92.92
0.73
79.24 + 0.36
FP6
HALO-1
66.54 0.22
93.56
0.38
80.20
0.26
HALO-2
67.42
0.99
93.56
0.38
80.00
0.62
HALO-0
4.50 1.03
55.9 03.98-2 + 20.89
74.73 : 0.45
INTS
HALD-1
62.27 0.64
93.23 0.45
0.59
HALO-2
68.15 + 0.08
93.79 0.08
80.12 0.31
A.9 HALO Inference Speedups
We present HALO as a www-precision fine-tuning method. However, it can be presented as QAT scheme where inference of the fine-tuned model will be done in low precision. To this end, following QuaRot [3]. we fuse the Hadamard transformations into the previous linear modules and just apply two Hadamards before out-projection and fown-projectio layers the Attention and MLP modules. Table shows the inference speedups of single Transformer block when

---

use HALO for INT8 fine-tuning. As expected, the speedups increase with batch size, peaking at batch size However, with batch size of 16, the multi-head attention module becomes another bottleneck, leading to reduced speedup gains.
Table 6: Inference runtimes of one Transformer block in LLAMA3-80 Model, fine-tuned with HALO- 2(F "p G² with different batch sizes (BS). We use 512 sequence length.
BS
BFI6
HALO
Speedup
3.21ms
2.20ms
1.46x
4
6.19ms
3,28ms
1,89
13.12ms
6.88ms
1.91
16
26.52ms
14.32ms
1.85x
A.10
Transformer Block Speedups
INT8
FP8
BS=4
BS=8
BS=16
BS=32
BS=4
BS=8
BS=16
BS=32
Ideal
1.62x
1.69
1.75x
1.70x
1.32x
1.37x
1.35x
1.28x
HALD-0
0
1.52x
1.63 x
1.67x
1.63x
1.27x
1.32x
1.32x
1.24x
HALO-1
1,26x
1.43x
1.51x
1.50x
1.07x
1.19x
1.22x
1.16x
HALD-2
1.15x
1.31x
1.36x
1.38x
0.99
1.11x
1.12x
1.10x
Table 7: HALO speedups for different batch sizes (BS) on three consecutive decoder blocks of LLAMA3-88 8B model with 512 sequence length on a single RTX 4090. Ideal shows the speedups when there is no quantization and Hadamard overheads. We bold the chosen scheme for each precision.
We evaluate using HALO on three consecutive LLAMAA-SB Transformer blocks (the largest. number of blocks that fit on one GPU with batch size 32). Table 7 shows the speedup numbers for different levels in HALO when we apply INT8 and FP8 precisions. Using INT8, the most accurate HALO level, HALO-2, achieves speedups of 1.15x to 1.38x compared to BF16 when using our kernels. The speedup increases with larger batch sizes, as the quantization and Hadamard overheads become less significant, and the matrix multiplications become the primary bottleneck. For FP8, we use HALO-0, which achieves speedups of 1.27 x to 1.32x, coming within 5% the ideal speedup. We note that since FP6 has the same TensorCore peak performance FP8 (with less read/write overhead), the speedups achieved with FP8 can be considered lower bound for the potential speedups with FP6 as well.
A.11 FP8 Linear Layer Speedup
We also benchmark HALO-O and HALO-1 with FP8 quantization Table HALO-O nearly on with ideal speedup, peaking 1.68x. We also provide HALO-1 results when we use with FP6 precision. However, since hardware support for FP6 matrix multiplication is unavailable, we use FP8 matmul instead to provide lower bound on the FP6 speedup, which peaks at 1.52x.
Table 8: Forward + backward speedups (over BF16) of linear layer (4096 4096) across batch sizes (BS) when we quantize inputs, weights, and output gradients using FP8 representation.
(RTX-4090)
BS=4
BS=8
BS=16
BS=32
Ideal
1.20x
1.65x
1.70x
1.78x
HALO-1 1
1.07
1.47x
1,48x
1.52x
HALO-O
1.12x
1.62x
1.63x
1.68x
A.12 HQ-FSDP Speedups
In Table we include detailed speedup numbers for FP8 HALO-0, FP8 HALO-1 and INT8 HALO-2 on four RTX 4090 GPUs, with and without HQ-FSDP.
25

---

w/o HQ-FSDP
w/ HQ-FSDP
BS=4
BS=8
BS=16
BS=32
BS=4
BS=8
BS=16
BS=32
FP8 Ideal
1.00x
1.01
1.06x
1.14x
1,39x
1.37x
1.31
1.28x
FP8 HALO-0
1.00
1.01x
1.05x
1.12x
1.39x
1.37x
1.30x
1.26x
FP8 HALO-1
0.98x
0.99x
1.02x
1.08x
1.37x
1.34x
1,25x
1.21
INT8 Ideal
1.00x
1.02x
1.14x
1.43x
1,40x
1.40
1.48x
1.65x
INT8 HALO-2
0.99x
1.00x
1.09
1.29
1.37x
1.34x
1.36x
1.45x
Table 9: HALO speedups for different batch sizes (BS) three consecutive decoder blocks of LLAMA3-8B model with 512 sequence length on four RTX 4090 GPUs, with and without HQ-FSDP. Ideal shows the speedups when there quantization and Hadamard overheads.
A.13 Qwen Results
To evaluate the effectiveness HAL.O on larger models, we extend our GSM8K INT8 experiments to the Qwen-2.5 14B and Qwen-2.5 32B models [44]. Table 10 compares HALO-2 against JetFire [41] Table
and the BF16 baseline. This table shows that HALO-2 outperforms both JetFire and BF16 across both model sizes.
Table 10: Comparison of HALO-2 with JetFire and BF16 on GSM8k for Qwen models.
Method
Qwen-14B
Qwen-32B
BF16
81.046 + 0.5
86.808 + 0.3
JetFire
81.92 = 1.12
87.1 = 0.71 88.40 + 0.8
HALO-2
82.33 0.68
A.14 Pre-training Results
We apply HALO to pre-training of TinyLlama-I.1 1. [47] on the C4 dataset [18]. We select a random Chinchilla-optimal subset [18] (22B tokens), and follow the same hyper-parameters as the original TinyLlama-1.1B [47]. Applying combinations of HALO levels and precisions (INT8, FP4, FP6, and FP8), our results can summarized follows:
FP8 closely matches the base BF16 training with HALO level 0, both achieving evaluation loss of 2.55.
FP6 converges only with HALO level 2, achieving final evaluation cross-entropy of 2.70.
INT8 and FP4 diverge, no matter the HALO level.
A.15 MXFP6 Results
In this section, we apply HALO-1 and HALO-2 on the MXFP6 format, comparing them with pure MXFP6 and BF16 training. We consider the LLAMA3-8Bmodel [13] and all three datasets mentioned in the main text. Table shows that HALO-2 closes the accuracy gap with BF16 on the GSM8k [8] dataset.
Table 11: MXFP6 fine-tuning results for LLAMA3-8B(13) on the three datasets, with and without HALO.
Method
GSM8k
ViGGO
SQL
BF16
69.3
94.0 0.3
79.9 0.5
MXFP6
67.8 0.2
93.6
79.6 1.0
MXFP6-HALO1
68.0
93.7 0.3
80.5 + 0.2
MXFP6-HALO2
0.59
93.5 0.5
80.3
26


Reviews:
[{"review_id": "Mk88JXbz2y", "reviewer": "Reviewer_pupA", "summary": "The authors introduce HALO for parameter-efficient fine-tuning (PEFT) of LLMs. HALO replaces traditional low-rank decomposition (as used in LoRA) with a structured Hadamard-based transformation. This transformation leads to efficient and more expressive representation of the update matrix, leading to reduced trainable parameters. The authors present results on varying tasks to prove the efficacy of the proposed method, comparing against existing PEFT methods like LoRA, AdaLoRA and QLoRA.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Idea is novel and simple to implement\n- Authors provide theoretical justification into HALO's expressiveness\n- The proposed idea is computationally efficient", "was_revised": true, "final_justification": "After reading the other reviews and the authors’ rebuttal, I maintain my original score.", "rebuttal": "We thank the reviewer for their comment on our work and would be happy to address any further questions or feedback."}, {"review_id": "YfmiaLv2x5", "reviewer": "Reviewer_C2CP", "summary": "The paper proposes HALO, a method for fine-tuning large language models (LLMs) using low-precision formats like INT8 and FP6. The key idea is to apply **Hadamard transformations** during both the forward and backward passes to reduce the impact of outliers in weights, activations, and gradients, which are major sources of instability in low-precision training. The method has good system support, including FSDP and HQ-FSDP. HALO achieves near full-precision accuracy on fine-tuning tasks like GSM8K, ViGGO, and SQL, while delivering up to **1.41× end-to-end speedup** on RTX 4090 GPUs.", "questions": "1. The paper shows that INT8 training with Hadamard transforms achieves efficiency comparable to FP8. Could the authors comment on how this approach compares with FP8 training in terms of stability, accuracy, and efficiency?\n2. Is INT4 training feasible with HALO? If not, what are the key challenges?\n3. Could the authors compare their method with related work such as RoLoRA [1], which also addresses rotated outlier mitigation for weight and activation quantization?\n\n[1] RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n\n1. The proposed method is well-motivated, targeting an important challenge in efficient LLM fine-tuning.\n2. The paper provides solid systems support, including integration with FSDP and quantized communication, which enhances its practical relevance.\n\n### Weaknesses:\n\n1. The novelty of the contribution is limited. The core technique of applying Hadamard transforms is a direct extension of prior work (e.g., QuaRot), and the application to training is relatively straightforward.\n2. The baseline comparisons are limited. Key existing methods such as LoRA, QLoRA, qDoRA, and qRoSA are not included, making it difficult to fully assess the empirical effectiveness of HALO.\n3. The presentation can be improved. The description of the method is currently organized as bullet points, which makes it harder to follow. A formal algorithm box would significantly improve clarity.\n4. Some promised results appear to be missing. For example, the paper mentions LoRA results on line 315, but I couldn't find them in the main text or appendix.", "was_revised": true, "final_justification": "I thank the authors for their responses. Most of my concerns have been addressed, and I have updated my score to a 4. However, I believe the clarity of the presentation can still be improved in the final version of the paper.", "rebuttal": "We thank the reviewer for their comment on our work. We address the questions here:\n\n> The novelty of the contribution is limited.\n\nWhile prior work has explored Hadamard transforms for inference, to our knowledge, HALO is the first to apply them across all three matrix multiplications in linear layers using INT8/FP6, while also delivering end-to-end training speedups. Achieving this involves two main challenges:\n\n- Search space complexity: Each matrix multiplication offers three potential insertion points for Hadamard transforms (see Section 3), leading to 8 configurations per matmul. For three matmuls per linear layer, this results in a combinatorially large design space.\n\n- Compatibility constraints: Exploring this space must remain compatible with precision-specific error analysis (Figures 1, 2), as well as with FSDP and activation checkpointing.\n\nHALO navigates these challenges by maintaining consistency across precision settings and supporting implementation constraints—such as applying identical transforms to weights in both forward and backward passes (crucial for HQ-FSDP) and enabling activation checkpointing (see Section 3.4). Efficient low-precision matmul and communication kernels back HALO’s design, yielding up to 1.4x training speedups on standard hardware.\n\n\n> The baseline comparisons are limited. Key existing methods such as LoRA, QLoRA, qDoRA, and qRoSA are not included, making it difficult to fully assess the empirical effectiveness of HALO.\n\nWe would like to emphasize that HALO is orthogonal to existing PEFT methods and can be combined with them. \nThis is demonstrated in Section A.3 of the Appendix. In particular, there we evaluated HALO combined with low-rank adaptation methods (referred to as HALO-PEFT) and presented the results in the same section. \nFor INT8 quantization, HALO-PEFT consistently remains within the standard deviation of the full-precision model fine-tuned with LoRA (rank r=16) on the GSM8K, ViGGO, and SQL tasks. \n\n> The presentation can be improved. The description of the method is currently organized as bullet points, which makes it harder to follow. A formal algorithm box would significantly improve clarity.\n\nThank you so much for your comment. We promise to add an Algorithm for each forward and backward passes for different HALO levels in the next revision of the paper\n\n> Some promised results appear to be missing. For example, the paper mentions LoRA results on line 315, but I couldn't find them in the main text or appendix.\n\nWe would like to note that the LoRA results were moved to the appendix due to space constraints. All results can be found in Appendix A.3.\n\n\n> The paper shows that INT8 training with Hadamard transforms achieves efficiency comparable to FP8. Could the authors comment on how this approach compares with FP8 training in terms of stability, accuracy, and efficiency?\n\nWe compare the results of applying HALO to INT8 with FP8 quantization without any Hadamard transformation (denoted as No-HALO) in Table 1 of Section 4.1. In our experiments, both approaches perform similarly and demonstrate stable behavior. Additionally, as shown in Table 2 of Section 4.2, we achieve comparable speedups of 35% on 4× GPUs and 39–41% on 8x GPUs for both INT8 (with HALO-2) and FP8 (with No-HALO). For pre-training results, please refer to the “General Response”.\n\n\n> Is INT4 training feasible with HALO? If not, what are the key challenges?\n\nOur experiments show that using the Hadamard transformation alone is insufficient to fit activation dynamic ranges into extremely narrow representable formats such as INT4. This issue can be addressed by tuning the clipping ratio during training, which is the focus of concurrent research [1].\n\n[1] https://arxiv.org/abs/2502.05003\n\n> Could the authors compare their method with related work such as RoLoRA, which also addresses rotated outlier mitigation for weight and activation quantization?\n\nWe thank the reviewer for sharing this related work. While RoLoRA employs a rotation-based approach, it focuses on weight and activation quantization, whereas errors (gradient of activations) are left unquantized. In contrast, HALO-PEFT fully quantizes all three components—weights, activations, and errors. For this reason, we believe RoLoRA is not a direct point of comparison with our method.\n\nAdditionally, we note that RoLoRA is a PEFT method, whereas HALO is designed primarily for low-precision full fine-tuning. That said, due to the conceptual similarity in the use of rotation-based techniques, we agree that a discussion of RoLoRA is warranted and will include one in the next revision of the paper."}, {"review_id": "qpi6v3J4iU", "reviewer": "Reviewer_bZTT", "summary": "The paper introduces HALO: a novel quantization approach that enables low-precision training by applying Hadamard transformations in the forward and backward passes of transformer architectures. The paper demonstrates the advantages of HALO in terms of end-to-end speedup for both single and multi-GPU LLM fine-tuning and parameter-efficient fine tuning at INT8 and FP6 precision.", "questions": "1) Can the authors elaborate on the advantages of HALO (and its variations) when it comes to total memory usage and energy consumption (e.g. flops). Are there any advantages according to these metrics? Do the additional Hadamard transforms introduce more computational overhead when compared to higher bit width precision training? \n\n2) Table 1 and Table 2 report the results for No-HALO at FP8. Are the NO-HALO results equivalent to vanilla low-bit width training? How do HALO-1 and HALO-2 perform in these settings? \n\n3) No experiment in the main text reports a comparison between HALO-1 and HALO-2. How do the two methods compare in terms of training accuracy vs compute?", "limitations": "The paper focuses on assessing speedup resulting from the HALO method, but a high-level analysis or discussion on power consumption is quite relevant considering the scale at which these method can be applied to train LLMs.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths \n\n1) Very clear and well-written paper. Strong structure, motivation and analysis \n\n2) The paper tackles a very prominent problem with potential of enabling more efficient LLM training and fine-tuning, \n\n3) The paper provides a solid evaluation and implementation, which make this work very easy to apply to existing LLMs. \n\n4) To the best of my knowledge, the paper first tackles the problem of reducing quantization artifacts in the gradients using invertible transformation, proposing an elegant solution with solid foundations. \n\n## Weaknesses\n1) While the methodology and analysis are very clear, the experimental section in the main text seems quite limited. The reported results focus on the trade-off between model accuracy and speed-up factors. A more diverse set of metrics would further strengthen the submission. \n\n   * The paper mentions other factors such as memory and gradient communication overhead. However, these metrics are not included in the experimental section. \n\n   * Hadamard transformations introduce additional (limited) overhead due to the extra un-fused operations. No reported comparison considers metrics focusing on compute such as Floating-point operations (or Bit Operations), which would make the comparison with baselines even more clear.  \n2) The paper clearly denotes the challenge of determining where each hadamard transform should be applied and provides 2 solutions: HALO-1 and HALO-2. However, the justification of which Hadamard transform should be applied (left, right or middle) is quite compressed and difficult to unpack in the main text. Figure 3 left could use some additional details to help the reader.", "was_revised": true, "final_justification": "The authours ' response is adequate and insightful. I confirm my recommendation for acceptance", "rebuttal": "We thank the reviewer for their comment on our work. We address the questions here:\n\n\n> the experimental section in the main text seems quite limited. The reported results focus on the trade-off between model accuracy and speed-up factors. A more diverse set of metrics would further strengthen the submission.\n\nGiven that our experiments center on supervised fine-tuning in lower precision, it appears natural to center on the trade-off between accuracy and speed-up. In addition, we have also examined the impact of transforms on outliers (e.g., Figure 1 and Appendix A.4). \nWe would be happy to provide any specific additional metrics that the reviewer finds relevant, during the discussion period.  \n\n\n> The paper mentions other factors such as memory and gradient communication overhead. However, these metrics are not included in the experimental section.\n\nWe can reduce memory usage by storing quantized activations during the forward pass and reusing these quantized tensors in the backward pass. This is particularly beneficial for large batch sizes or long input sequences. Specifically, consider a single linear layer of size $d_1 \\times d_2$​, with batch size $b$ and sequence length $t$; activation memory becomes a bottleneck when $bt \\ge d_2$.\n\n\n\n> Hadamard transformations introduce additional (limited) overhead due to the extra un-fused operations. No reported comparison considers metrics focusing on compute such as Floating-point operations (or Bit Operations), which would make the comparison with baselines even more clear.\n\n\nWe illustrate the overhead of applying the Hadamard transformation in Figure 4b by comparing HALO-2 with the \"Ideal\" case—an idealized speedup scenario where no Hadamard transformation is applied and quantization is simply performed by casting from BF16 to INT8 across three consecutive transformer blocks. The results show that, in the worst case, the combined overhead of the Hadamard transformation and quantization function accounts for less than 14% of the total runtime when using our custom kernels (excluding FSDP quantization).\n\n> the justification of which Hadamard transform should be applied (left, right or middle) is quite compressed and difficult to unpack in the main text. Figure 3 left could use some additional details to help the reader.\n\nWe appreciate your feedback and fully agree with your observation. We will address this in the next revision.\n\n> Can the authors elaborate on the advantages of HALO (and its variations) when it comes to total memory usage and energy consumption (e.g. flops). Are there any advantages according to these metrics?\n\nPlease check the second and third questions above, as well as the last question below.\n\n\n> Do the additional Hadamard transforms introduce more computational overhead when compared to higher bit width precision training?\n\n\nPlease check the third question above.\n\n\n\n> Table 1 and Table 2 report the results for No-HALO at FP8. Are the NO-HALO results equivalent to vanilla low-bit width training? How do HALO-1 and HALO-2 perform in these settings?\n\nYes. While it is possible to apply HALO levels in this case (without causing training instability), we found that FP8 does not require such transformations. Therefore, introducing the additional overhead of applying them is unnecessary in the FP8 setting.\n\n\n> No experiment in the main text reports a comparison between HALO-1 and HALO-2. How do the two methods compare in terms of training accuracy vs compute?\n\nThese experiments are included in the Appendix. For the accuracy comparison, please see Table 5 in Appendix A.8. For the speedup results, refer to Table 7 in Appendix A.10.\n\n> The paper focuses on assessing speedup resulting from the HALO method, but a high-level analysis or discussion on power consumption is quite relevant considering the scale at which these method can be applied to train LLMs.\n\nWe thank the reviewer for raising this valuable point. Power consumption typically arises from two main sources: data movement and computation.\n\n- Data movement: Although HALO maintains master copies of weights, activations, and gradients in BF16, it stores quantized versions of weights and activations for the backward pass, which partially reduces data transfer costs. As for the Hadamard transforms, while they do introduce additional reads, we believe that in an optimized implementation where these operations are fused, this overhead can be significantly reduced.\n\n- Computation: All matrix multiplications in HALO are performed in low precision, which substantially reduces compute-related power usage. The computation involved in the Hadamard transforms is minimal compared to that of the matrix multiplications and thus contributes negligibly to overall power consumption.\n\nGiven these factors, we expect the speedups reported in the paper to correlate strongly—though not perfectly—with reductions in power consumption. We appreciate the reviewer’s suggestion and will include a discussion of this point in the next revision."}, {"review_id": "nD00rwbM3K", "reviewer": "Reviewer_iAzw", "summary": "This paper proposes HALO, a quantized LLM fine-tuning framework in which all large matrix multiplications, including those in the backward pass, are performed in low precision. Based on the analysis that quantization in the forward path has a significantly greater impact on gradient error, HALO mitigates this by multiplying Hadamard matrices to the right of the input $X$ and the weight $W$. For more aggressive quantization, such as INT8, HALO-2 further applies the Hadamard matrix to the output error. Since HALO quantizes the majority of tensors during fine-tuning, it reduces communication and activation checkpointing costs. HALO achieves a 1.41× speedup for INT8 fine-tuning and also outperforms existing low-bit training frameworks.", "questions": "1. Regarding Weakness 1, can the authors compare the efficiency of HALO and LoRA? Specifically, does HALO provide faster fine-tuning speed than LoRA?\n2. Table 2 and the text give the impression that both FP8 without HALO and INT8 with HALO are considered part of the proposed scheme. It is unclear what modifications or additions have been made for FP8 No-HALO compared to naïve weight, activation, and gradient quantization. Could the authors clarify this?\n3. Could the authors discuss the applicability of HALO to the pre-training setup and provide a comparison with Jetfire in that context?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The proposed method is grounded in a solid observation, providing a valuable knowledge contribution.\n2. The paper addresses practical and common fine-tuning scenarios, such as FSDP and gradient checkpointing, and the proposed method appears to integrate well with these techniques.\n\n**Weaknesses**\n\n1. The motivation for quantized fine-tuning is unclear. While it is true that quantization-aware fine-tuning is more challenging than quantization-aware training from scratch due to outliers, there are many viable alternatives to full fine-tuning with quantization, such as LoRA, Prompt Tuning (Lester et al., 2021), and Side Ladder Tuning (Sung et al.). Comparing the results in Table 1 and Table 4 suggests that HALO’s full fine-tuning does not demonstrate a clear accuracy advantage over LoRA, which may limit its adoption compared to other methods.\n2. The paper’s scope is limited to fine-tuning, although the proposed technique could also be applied to the more resource-intensive pre-training stage. Discussing training performance on GPT2-sized models and smaller datasets like WikiText-103 could increase the paper’s impact.\n3. The overall contribution is marginal. Identifying a major source of quantization error is a valuable addition that will be appreciated by the ML community. However, other contributions—such as multiplying Hadamard matrices before quantization in Steps 2 and 3 of Section 3.2—are somewhat trivial, given that similar techniques have been adopted in QuIP# and QuaRot. The GPU kernel implementation is a positive aspect, but the speedup reported from Jetfire is not very significant.\n\nLester et al. “The power of scale for parameter-efficient prompt tuning.” *EMNLP 2021*.\n\nSung et al. \"Lst: Ladder side-tuning for parameter and memory efficient transfer learning.” *NeuRIPS 2022.*", "was_revised": true, "final_justification": "My concerns were resolved by the authors' rebuttal. Specifically, the significance of the proposed method compared to LoRA was well addressed. I thank the authors again for providing the sufficient materials for the reassessment.", "rebuttal": "We thank the reviewer for their comment on our work. We address the questions here:\n\n\n> The motivation for quantized fine-tuning is unclear. While it is true that quantization-aware fine-tuning is more challenging than quantization-aware training from scratch due to outliers, there are many viable alternatives to full fine-tuning with quantization, such as LoRA, Prompt Tuning (Lester et al., 2021), and Side Ladder Tuning (Sung et al.). Comparing the results in Table 1 and Table 4 suggests that HALO’s full fine-tuning does not demonstrate a clear accuracy advantage over LoRA, which may limit its adoption compared to other methods.\n\n> Can the authors compare the efficiency of HALO and LoRA? Specifically, does HALO provide faster fine-tuning speed than LoRA?\n\nWe present HALO primarily to accelerate full fine-tuning (FT) which is computationally expensive, as each linear layer requires three full matrix multiplications during both the forward and backward passes. We would like to emphasize the following points:\n\n- Full fine-tuning remains a highly important approach. Prior work [1] has shown that PEFT methods (such as LoRA) underperform compared to full fine-tuning on various tasks, including the HumanEval benchmark [2], even when using large ranks.\n- HALO can also be integrated into LoRA-style fine-tuning without accuracy drop. In the paper, we demonstrate this as HALO-PEFT (see Appendix A.3), where the most expensive parts of LoRA are executed in low precision.\n- Finally, we emphasize that HALO can outperform LoRA in terms of speedups. While LoRA benefits from skipping one full matrix multiplication in the backward pass—since the base weights are frozen and their gradients are not computed—this leads to a speedup of up to ~30% in full fine-tuning (if we skip the overhead of low-rank computations). In contrast, HALO-2 can accelerate a single linear layer by up to 80%, as shown in Figure 4 of the paper.\n\n[1] https://arxiv.org/pdf/2405.09673\n\n[2] https://arxiv.org/pdf/2107.03374\n\n> The paper’s scope is limited to fine-tuning, although the proposed technique could also be applied to the more resource-intensive pre-training stage. Discussing training performance on GPT2-sized models and smaller datasets like WikiText-103 could increase the paper’s impact.\n\n> Could the authors discuss the applicability of HALO to the pre-training setup and provide a comparison with Jetfire in that context?\n\nHALO can be applied during pre-training without any modifications, achieving the same speedups. We demonstrate this by applying HALO to the pre-training of TinyLlama-1.1B [3] on the C4 dataset [4], using a randomly selected Chinchilla-optimal subset of 22B tokens and hyperparameters similar to those in [3]. our results can be summarized as follows:\n- FP8 closely matches the base BF16 training even with No-HALO, both achieving an evaluation loss of 2.55.\n- FP6 converges only with HALO-2, achieving a final evaluation cross-entropy of 2.70.\n- INT8 and FP4 diverge, no matter the HALO level.\n\n[3] https://arxiv.org/abs/2401.02385\n\n[4] https://arxiv.org/abs/2203.15556\n\n> The overall contribution is marginal. Identifying a major source of quantization error is a valuable addition that will be appreciated by the ML community. However, other contributions—such as multiplying Hadamard matrices before quantization in Steps 2 and 3 of Section 3.2—are somewhat trivial, given that similar techniques have been adopted in QuIP# and QuaRot.\n\nWe would also like to highlight that HALO differs from prior rotation-based quantization schemes (such as QuaRot), as the computational invariance trick used in those methods does not apply when both the forward and backward passes are involved during training. We provide a detailed explanation in Section 3.1 and Appendix A.1.\n\n> The GPU kernel implementation is a positive aspect, but the speedup reported from Jetfire is not very significant.\n\nWe would like to emphasize that, although we achieve approximately 20% speedup compared to JetFire, implementing an efficient kernel for group-wise quantization without significant overhead is challenging and often highly model-dependent. Our reported ~20% speedup is based on the simplest quantization scheme—AbsMax tensor-wise quantization.\n\n\n> Table 2 and the text give the impression that both FP8 without HALO and INT8 with HALO are considered part of the proposed scheme. It is unclear what modifications or additions have been made for FP8 No-HALO compared to naïve weight, activation, and gradient quantization. Could the authors clarify this?\n\nSince FP8 offers a sufficiently wide dynamic range (with 4 or 5 exponent bits), models can be fine-tuned without applying any Hadamard transformation. This corresponds to a naive FP8 quantization scheme, where quantization is applied directly to both operands before the matrix multiplication, using a low-precision MatMul kernel. We refer to this approach as “No-HALO” in the paper, as it does not fall under the HALO family of methods.\n\nHowever, it's important to note that FP8 is not supported on all hardware accelerators (especially commodity GPUs). In such cases, HALO-1 and HALO-2 can achieve comparable accuracy (see Table 1, Section 4.1) and similar end-to-end speedups (see Table 2, Section 4.2) using other precision formats such as FP6 and INT8."}, {"review_id": "13rV1Tuytl", "reviewer": "Reviewer_mshD", "summary": "This paper introduces HALO (Hadamard-Assisted Lower-Precision Optimization), a novel training strategy that enables fully quantized fine-tuning of large language models (LLMs) using INT8 and FP6 precision for all major matrix multiplications in both forward and backward passes. The method strategically inserts Hadamard transformations to mitigate quantization-induced outliers and integrates optimized low-precision CUDA kernels and FSDP-based communication. The authors demonstrate that HALO achieves near-lossless accuracy compared to BF16 baselines across multiple tasks (GSM8K, ViGGO, SQL) and models (LLaMA3-8B, Qwen), while achieving significant speedups on commodity GPUs.", "questions": "1. Could HALO be extended to other orthogonal transforms such as DCT or Givens rotations? Are there trade-offs in accuracy or compute cost?\n2. Can HALO be combined with structural model compression (e.g., pruning, distillation) for further efficiency?", "limitations": "1. The method focuses solely on outlier suppression using Hadamard rotations, without addressing other sources of quantization error such as sparsity or skewed distributions.\n2. The overhead of Hadamard transforms is more significant when batch size is small or when tensor dimensions are not power-of-two.\n3. The current implementation relies on tensor-wise quantization; the interaction with finer-grained schemes (e.g., per-channel or block-wise) remains unexplored.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths:\n1. HALO introduces an elegant use of orthogonal Hadamard transforms during training to suppress outliers in activations, weights, and gradients, extending ideas from inference quantization (e.g., QuaRot) into the more challenging fine-tuning setting.\n2. The method outperforms previous quantized fine-tuning baselines (e.g., SwitchBack, Jetfire) under INT8 and FP6 with minimal accuracy degradation (<1%), and yields up to 1.41× speedup.\n3. Efficient CUDA-based Hadamard and quantized matmul kernels are implemented, with careful integration into CUTLASS and FSDP/activation checkpointing frameworks.\n4. HALO supports both full fine-tuning and parameter-efficient fine-tuning (LoRA), and is tested on different families of LLMs.\n\n### Weaknesses:\n1. All benchmarks are run on RTX 4090; it is unclear how well HALO performs on data center GPUs (A100, H100) or non-NVIDIA architectures.\n2. The approach is validated only for fine-tuning; its applicability to pretraining or continued training from scratch is not assessed.\n3. The paper does not explore extending HALO to INT4 or 4-bit floating point formats.", "was_revised": true, "final_justification": "I now view this as a solid systems-methods contribution that will benefit both researchers and practitioners working on scalable, low-precision LLM training.", "rebuttal": "We thank the reviewer for their comment on our work. We address the questions here:\n\n> All benchmarks are run on RTX 4090; it is unclear how well HALO performs on data center GPUs (A100, H100) or non-NVIDIA architectures.\n\nWe have chosen to focus on the 4090 due to the fact that it is a recent GPU and has hardware INT8 support. By contrast, the A100 is already two generations old, and the H100 only has FP8 support (and not INT8 or INT4). \nThe newer line of Blackwell RTX GPUs (RTX 5090, RTX 6000 Blackwell) does have support for INT8, meaning that our results will continue to be relevant on current hardware, especially since the latter is a datacenter GPU. In addition, the AMD CDNA (MI series) MI100, MI200, MI300X GPUs all support INT8 matrix multiply via their Matrix Core Engines (MCEs). They do not support INT4 or FP8 precisions, to our knowledge. \n\n> The approach is validated only for fine-tuning; its applicability to pretraining or continued training from scratch is not assessed.\n\nHALO can be applied during pre-training without any modifications, achieving the same speedups. We demonstrate this by applying HALO to the pre-training of TinyLlama-1.1B [3] on the C4 dataset [4], using a randomly selected Chinchilla-optimal subset of 22B tokens and hyperparameters similar to those in [3]. our results can be summarized as follows:\n- FP8 closely matches the base BF16 training even with No-HALO, both achieving an evaluation loss of 2.55.\n- FP6 converges only with HALO-2, achieving a final evaluation cross-entropy of 2.70.\n- INT8 and FP4 diverge, no matter the HALO level. We believe there is additional work needed on stabilizing these lower precisions for pre-training, especially in the earlier chaotic period. \n\n[3] https://arxiv.org/abs/2401.02385\n\n\n[4] https://arxiv.org/abs/2203.15556\n\n> The paper does not explore extending HALO to INT4 or 4-bit floating point formats.\n\n\nOur experiments show that using the Hadamard transformation alone is insufficient to fit activation dynamic ranges into extremely narrow representable formats such as INT4 of FP4. In our experiments, the FP4 and INT4 cases were diverged. \n\n\n> Could HALO be extended to other orthogonal transforms such as DCT or Givens rotations? Are there trade-offs in accuracy or compute cost?\n\nWe thank the reviewer for this insightful question. Yes, it is possible to use transformations other than the Hadamard transform. However, the chosen transformation should (1) be fast enough to provide a meaningful speedup when applied before quantized matrix multiplication, and (2) effectively address outliers to ensure training stability. The Hadamard transform satisfies both criteria and benefits from an efficient CUDA kernel, being less than 1.8× slower than a memcpy operation (which involves only reading and writing) [1].\n\n[1] https://github.com/Dao-AILab/fast-hadamard-transform \n\n\n> Can HALO be combined with structural model compression (e.g., pruning, distillation) for further efficiency?\n\nHALO is orthogonal to other model compression techniques and can be combined with them to further improve efficiency. More specifically, post-training compression methods can be applied either before or after HALO fine-tuning without any impact, as HALO does not modify the model architecture. Similarly, training-aware compression schemes can be used before or after the Hadamard transformations and integrated with HALO.\n\n\n> The method focuses solely on outlier suppression using Hadamard rotations, without addressing other sources of quantization error such as sparsity or skewed distributions.\n\nWe acknowledge this point, but please note that 1) standard normalization layers (e.g. RMSNorm) already reduce distribution skew significantly, and 2) handling of outliers via unstructured sparsity (e.g. via sparse-quantized formats, which we believe the reviewer is alluding to) are extremely difficult to support efficiently in hardware and would therefore not yield any practical speedups. \nBy contrast, our HALO design shows that near-lossless results can be achieved via INT8 quantized training even with limited but highly-efficient outlier suppression techniques.  \n\n\n> The overhead of Hadamard transforms is more significant when batch size is small or when tensor dimensions are not power-of-two.\n\n\nWe would like to highlight the following points:\n1. We agree that the Hadamard transformation introduces more overhead for extremely small batch sizes. However, in such cases, training often becomes memory-bound, and quantizing both weights and activations for low-precision matrix multiplications can help. In this scenario, the quantization operation itself becomes a bottleneck, as it requires two reads (to find the maximum and scale the values) and one write.\n2. For non-power-of-two cases, one can use grouped Hadamard transformations with the largest power-of-two size that divides the original size, without significantly sacrificing accuracy [1].\n\n[1] https://arxiv.org/abs/2502.05003\n\n\n> The current implementation relies on tensor-wise quantization; the interaction with finer-grained schemes (e.g., per-channel or block-wise) remains unexplored.\n\nWe emphasize tensor-wise quantization because it is the simplest scheme in terms of kernel implementation, introducing almost no overhead during training. Although finer-grained quantization schemes can improve model accuracy by reducing quantization error, they also add overhead that makes practical implementation more challenging. \nThis contrast is evident in our comparison with JetFire, which is a block-wise quantization technique."}]

Author rebuttal:

Meta-review: The rebuttal and author discussion process were quite successful. The new pre-training results and the clear description of HALO's novelty convinced all the reviewers, even the initially skeptical ones. Several reviewers raised their scores, leading to a strong consensus for acceptance. The final consensus was that HALO is a solid and practical method that will be valuable for researchers and practitioners.
Decision: Accept (poster)