To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability

## Abstract

Abstract The massive computational costs associated with large language model (LLM) pretraining have spurred great interest in reduced-precision floating-point representations to accelerate the process. As a result, the BrainFloat16 (BF16) precision has become the de facto standard for LLM training, with hardware support included in recent accelerators. This trend has gone even further in the latest processors, where FP8 has recently been introduced. However, prior experience with FP16, which was found to be less stable than BF16, raises concerns as to whether FP8, with even fewer bits than FP16, can be a cost-effective option for LLM training. We argue that reduced-precision training schemes must have similar training stability and hyperparameter sensitivities to their higher-precision counterparts in order to be cost-effective. However, we find that currently available methods for FP8 training are not robust enough to allow their use as economical replacements. This prompts us to investigate the stability of reduced-precision LLM training in terms of robustness across random seeds and learning rates. To this end, we propose new evaluation techniques and a new metric for quantifying loss landscape sharpness in autoregressive language models. By simulating incremental bit reductions in floating-point representations, we analyze the relationship between representational power and training stability with the intent of aiding future research into the field.

## 1 Introduction

Conversational large language models (LLMs), such as ChatGPT [], Gemini [, ], Claude [], Llama [, ], and HyperCLOVA [], have captured the imagination of both academics and the public at large with their ability to communicate fluently with humans in natural language. However, these models require unprecedented amounts of computation to train, which has engendered interest in methods to improve training efficiency.

A popular method of improving computational performance is to reduce the bit count of the floating-point representations used for training [, , ]. This is because reading memory is the main bottleneck in modern processors, a problem known as the “memory wall” [, ]. Reducing the number of bits that each floating-point number uses can, therefore, accelerate the computation in proportion to the amount of memory reduced. For example, in processors that support it, computations in BrainFloat16 (BF16) [] can have double the maximum throughput of single precision FP32. Furthermore, the FP32 data type, the highest precision data type used in deep learning, has only half the bits of FP64, the most widely used floating-point data type in scientific computing. The current best practice for LLM training is to use BF16 for most of the LLM training computation, with some sensitive portions, such as layer normalization [], carried out in FP32.

As a natural extension of this development, 8-bit floating-point (FP8) [, , , ] and even 4-bit floating-point [] data formats have been proposed to accelerate training even further. However, the naïve application of FP8 to LLM training is unstable and requires additional techniques to become viable. While several methods have been proposed to stabilize training LLMs with FP8, relatively little attention has been paid to quantifying the decrease in training stability compared to the standard mixed-precision BF16 training.

As the motivation behind the use of FP8 and other reduced-precision training schemes is cost reduction, our concern here is not whether LLM pretraining with FP8 is possible but whether it is profitable. For cost savings to be realized, the time per training step must be reduced while the number of training steps is kept to a similar number. Training stability is thus a crucial factor for cost-effective LLM training, considering that additional hyperparameter searches and restarts from training failures can outweigh any gains from the acceleration in raw computation. Therefore, for the newly proposed reduced-precision training schemes to be economical, the models trained on them must be similarly robust to hyperparameter choice and stochastic noise as models trained using higher precision.

Previous experience with training LLMs in FP16 raises further concerns. Teams that have trained LLMs have found that even when gradient scaling and other techniques are applied, the FP16 data type, which has five exponent bits, is much less stable for LLM training than BF16, which has eight exponent bits as in FP32. This raises doubts as to whether FP8, which has even fewer bits than FP16, is a practical option for real-world LLM training.

We motivate our line of inquiry with some surprising findings from experiments on the nanoGPT [] codebase, an open-source implementation of GPT-2 pretraining, where we found that even the current best practice of mixed-precision BF16 can introduce training instabilities. When we compared BF16 and TensorFloat32 (TF32) runs, where we ran training for 5% of the dataset for each run, we found that the BF16 models diverged for 18 of 188 runs, or approximately 10% of all cases, despite using the same configurations as the default run. In contrast, no cases of loss divergence were found for the 70 TF32 models that were trained using different random seeds. We compare against the TF32 data type because NVIDIA GPUs do not offer tensor cores in FP32.

This is a surprising finding in light of the fact that most recent LLMs are trained with mixed precision BF16 without a comparison with training on TF32, which has three additional mantissa bits. However, a loss divergence rate of approximately 10% at only 5% of training indicates that even standard BF16 mixed-precision training may add non-trivial instability. If even mixed-precision BF16 can cause instabilities, the effects of using even fewer bits should be investigated further.

We make the following contributions in our work.

- •
We analyze the effects of incremental bit reduction on training stability by simulating intermediate-bit representations of floating-point numbers. As observed in Kalamkar et al. [], we confirm that reducing exponent bits has a greater impact on model performance than reducing mantissa bits, finding that removing even a single exponent bit on all layers precludes training altogether for Llama models.
- •
We propose a metric for quantifying the loss landscape sharpness of models that can predict when training divergence will occur. As even the removal of mantissa bits has a destabilizing effect on LLM training, we use our metric to predict loss divergences even when the loss curve itself has not yet diverged.

Figure: Figure 1: We show three cases of loss divergence on nanoGPT when using the same configurations as the default run except for the random seed. The blue lines indicate the average losses obtained for eight training runs that did not diverge. Of the 188 random seeds that were tested, 18 were found to diverge. As full pretraining requires over 4 days on a single node with 8 A100 GPUs, even for BF16, we perform early stopping at 30K steps, or 5% of the original training runs, requiring approximately 4 hours for a BF16 run and 8 hours for a TF32 run per A100 node with 8 GPUs. Because we only use 5% of the training data, we suspect that the measured divergence rate of approximately 10% underestimates the true rate of training loss divergence.
Refer to caption: extracted/5628109/images/nanogpt-seed-15940.png

## 2 Related work

### 2.1 Training stability

Analyzing training instability in LLM pretraining directly is impractical due to the massive costs involved. Instead, smaller language models must be used as proxies. Wortsman et al. [] explore the robustness of smaller language models as a proxy for LLM pretraining instability. They find that small models using a larger learning rate show similar instability patterns as larger models, showing unstable behavior, such as the growth of attention layer logits and the divergence of the output logits from the log probabilities, can be reproduced in smaller Transformers. They also explore both the causes of numerical instability in LLM training and mitigating strategies such as applying query/key (QK) normalization [].

Keskar et al. [] explores the sharpness of loss landscapes during large-batch neural network training, finding that larger batch sizes prevent the model from reaching flat regions of the loss landscape and causing performance degradation due to the inability to escape local minima. Of most significance to our work, they propose a metric for calculating loss landscape sharpness, which we adapt for LLMs.

### 2.2 Reduced-precision processors

To improve throughput on computationally intensive matrix multiplication tasks, recently developed processors have been equipped with specialized hardware units such as systolic arrays for TPUs [] and tensor cores [], which serve a similar purpose, for NVIDIA GPUs. These features can improve throughput by an order of magnitude. For example, for BF16, the maximum throughput on tensor cores is 312 TFLOPS compared to 39 TFLOPS using CUDA cores [].

However, the number of multiplexer circuits required for the barrel shifter of an $n$-bit floating-point unit is $n\log_{2}n$ [], which incentivizes the use of smaller floating-point representations. As a result, many mixed-precision techniques perform computationally intensive matrix multiplication tasks in BF16 while preserving sensitive portions of the model, such as the weights and residual path activations, in FP32. Alternatively, Henry et al. [] developed a technique to approximate FP32 matrix multiplication using only BF16 by representing a single FP32 value as three BF16 values, which could accelerate FP32 matrix multiplication without requiring FP32 circuits.

### 2.3 Hybrid FP8

The adoption of the hybrid E5M2/E4M3 formats for neural networks [] in recent generations of processors, such as the NVIDIA Hopper series of GPUs and the Intel Gaudi v2, has spurred interest in stable FP8 training. The hybrid FP8 format, where E4M3 is used for the forward pass for its greater resolution and E5M2 is used for the backward pass due to its greater expressive range, was first proposed by Wang et al. [] as a means to accelerate both the training and inference stages of neural networks.

Sun et al. [] built on this work to propose various techniques for stabilizing training, such as stochastic rounding, chunk-based accumulation, and Kahan summation during the optimizer update. However, the number of techniques that can be used in practice is limited by whether the technique in question can be applied without slowing the computation. As current NVIDIA GPUs do not support these techniques natively, the overhead caused by the software-based implementations cancels out any gains from the reduced precision.

### 2.4 MS-AMP

Introduced in Peng et al. [], MS-AMP is an automatic mixed-precision package to utilize FP8 that offers multiple optimization levels to allow for differing model sensitivities when applying reduced precision to the computations and communications of neural network training. Our experiments use the O1 optimization level of MS-AMP, which performs GEMM computations in FP8 while reducing the weights to half-precision and uses FP8 states for the weight gradients and all-reduce communications. MS-AMP offers additional optimizations for the optimizer buffer in level O2 optimization and for distributed communication in level O3 optimization, but we choose to use only the most basic optimization scheme so as to verify the effects of the least invasive modifications.

## 3 Methods

We seek to answer whether sub-BF16 training is competitive with standard mixed-precision BF16 training from a cost-effectiveness point of view. In order to be cost-effective, reduced-precision training schemes must have minimal increases in training instability and changes to hyperparameters. To better analyze the effect of reduced precision on training stability, we aim to quantify the effects of gradually reducing floating-point representations bit by bit for both the exponent and the mantissa. Hopefully, analyzing the intermediate bit representations will better illuminate the interaction between bit width and training stability.

Our intermediate-bit floating-point experiments use the TinyLlama [] repository, which implements the widely used Llama [, ] architecture. TinyLlama is an open-source implementation of Llama pretraining that uses a mix of the SlimPajama [] and StarCoder [] datasets. It also includes performance optimizations such as Flash Attention v2 []. We use the default learning rate of $lr=4e-4$, global batch size 512, and the same learning rate schedule as in the original code. The 120M models use a sequence length of 2048, while the 7B models use a sequence length of 4096.

### 3.1 Sharpness metric

Figure: Figure 2: Loss landscape diagrams for Llama 120M E8M3 at 5K steps (left) and 10K steps (right). Even during loss divergence, the loss landscape visualized using the method in [] appears smooth, motivating our introduction of a new loss landscape sharpness metric. The value of the validation loss is included at each point of the loss landscape.
Refer to caption: extracted/5628109/images/heatmap_llama120m_e8m3_5k.png

To better investigate the state of the model when loss divergence occurs, we attempted to visualize the loss landscape of the Llama models with the method proposed by Li et al. [] in Figure 2. However, we found that even when the model is clearly in the process of loss divergence, the generated visualizations do not show signs of sharpness. Because of this, we propose a modified loss landscape sharpness metric that is more suitable for autoregressive models based on the one proposed in Keskar et al. []. The main difference with our version is that we use the logit of the last token instead of the model input for the calculation. This is because adding noise to the embeddings in a language model has different implications compared to adding noise to input images in a computer vision model. Instead of searching the input space as in Keskar et al. [], we apply the search algorithm to the logit space of the last token. Searching the logit space has the additional advantage that the forward pass of the model need only be performed once for each measurement, greatly reducing the computational cost. The logit of the last token is chosen because it is not computationally feasible to perform an optimization on the entire output space. Also, due to the autoregressive character of decoder-only generative language models, the last token is the only one to receive inputs from all other tokens.

#### Definition

Let $\boldsymbol{y}\in\mathbb{R}^{s\times v}$ be the output logit for an autoregressive model of sequence length $s$ and vocabulary size $v$. Then, for $\boldsymbol{y}_{i}$, the output logit at sequence position $i\in\{1,2,...,s\}$, and one-vector $\mathbf{1}_{v}\in\mathbb{R}^{v}$, we define a constraint set $\mathcal{C}_{\epsilon}$ at $i=s$ such that

$$ $\mathcal{C}_{\epsilon}\in\{\boldsymbol{z}_{s}\in\mathbb{R}^{v}:-\epsilon(| \boldsymbol{y}_{s}|+\mathbf{1}_{v})\leq\boldsymbol{z}_{s}\leq\epsilon(| \boldsymbol{y}_{s}|+\mathbf{1}_{v})\}.$ (1) $$

Given $\boldsymbol{y}_{s}\in\mathbb{R}^{v},\epsilon>0$, and noise vector $z_{s}$, the loss landscape sharpness $\phi_{\epsilon}$ for loss function $f$ can be defined as

$$ $\phi_{\epsilon}:=\frac{max_{\boldsymbol{z}_{s}\in\mathcal{C}_{\epsilon}}f( \boldsymbol{y}_{s}+\boldsymbol{z}_{s})-f(\boldsymbol{y}_{s})}{1+f(\boldsymbol{ y}_{s})}\times 100.$ (2) $$

The proposed metric can best be thought of as the relative magnitude of the largest loss spike on the logit within the provided bounds. The bounds are set to be the logit magnitudes plus one multiplied by $\epsilon$. The largest spike in the vicinity of the logits is found using the L-BFGS-B algorithm [], using the SciPy [] implementation with the output logit set as the starting point of the search. We use $\epsilon=5e\mathrm{-4}$ for all our experiments. Also, loss sharpness is measured with validation data that has not been used for training. In addition, we do not apply random projection as in Keskar et al. [] to reduce the stochasticity of the measurement.

### 3.2 Masking

Figure: Figure 3: Diagram showing the precisions used in a Llama decoder block (best seen in color). The activations in the path of the residual connection are kept in FP32, as are the model weights and embeddings. The LayerNorm and RoPE layers use FP32 internally for their computations. The Flash Attention kernel uses BF16 with no reduction in precision. All other layers use reduced-precision matrix multiplication that emulates low-precision computation with a high-precision accumulator.
Refer to caption: extracted/5628109/images/precision_diagram.png

Figure: Algorithm 1 PyTorch-like pseudocode for the reduced-precision forward pass.
Refer to caption: extracted/5628109/images/masking_method.png

Our experiments use a simplified method of reducing floating-point precision to achieve reasonable throughput. To simulate removing exponent bits, we threshold the values to the minimum and maximum absolute values possible with the given number of exponent bits. Figure 4 depicts the exponent masking process. To remove the mantissa bits, a bitmask is applied to remove the unrepresentable bits. The resulting method is an imperfect approximation of reducing the bit count of floating-point numbers. However, it has the advantage of being fast at run time, causing at most a doubling of the time per each training step.

Reduced precision operations are applied only on the matrix multiplication computations of the model, excluding the attention computation, which uses the Flash Attention v2 kernel. Following existing FP8 libraries such as TransformerEngine [], we separate the effects of reducing the computation’s precision from reducing the data’s precision in storage. As a result, the activations and model weights are kept at their original precision while the inputs and outputs of matrix multiplication are dynamically masked to emulate reduced precision computation with a high-precision accumulator. All states are kept in their original precision, and all operations other than matrix multiplication are performed in their original precision. In Figure 3, we include a diagram indicating the precision of the states and computations in a Llama decoder block.

## 4 Results

### 4.1 MS-AMP experiments

We first analyze the effect of real-world FP8 training by applying the MS-AMP [] library (version 0.3.0) to the nanoGPT codebase. We run all experiments for MS-AMP on an H100 node with 8 GPUs to ensure hardware availability of FP8. However, despite only using the O1 optimization level for MS-AMP, we find that the resulting models show non-trivial performance degradations. These results indicate either that MS-AMP cannot converge to the same loss as BF16 training or that it requires more training steps. In either case, the justification for deploying FP8 training as a cost-saving measure becomes weaker.

Figure: Figure 5: Comparison between training with BF16 and with MS-AMP level O1. Experiments on MS-AMP were conducted on the nanoGPT codebase with only the model architecture as a variable. In both cases, even level O1 optimization of MS-AMP underperforms BF16 training noticeably. We use a sequence length of 1024 and a global batch size of 480 on a single node with eight H100 GPUs. The nanoGPT codebase was used because we were unable to integrate TinyLlama with MS-AMP.
Refer to caption: extracted/5628109/images/msamp_gpt2_124m.png

### 4.2 Bit reduction experiments

Figure: Figure 6: TinyLlama 120M models trained until loss divergence occurred. E8M3, E8M4, and E8M5 models trained for 16K, 20K, and 100K steps, respectively. The dotted black line in each figure indicates the loss landscape sharpness of the model. A global batch size of 512 was used with a sequence length of 2048.
Refer to caption: extracted/5628109/images/tinyllama120m_e8m3.png

Figure: Figure 7: Llama 7B model training loss curves for different mantissa bit masks. The x-axis represents training steps, while the y-axis represents the training loss.
Refer to caption: extracted/5628109/images/tinyllama7b_e8m3456.png

We first attempt to identify the points where training instability becomes visible. The emulated reduced-precision representations are denoted using the number of exponent and explicit mantissa bits used. For example, standard BF16 is referred to as E8M7, while a floating-point number with its exponent clamped to seven bits and mantissa clamped to six bits is referred to as E7M6.

We found that removing even a single exponent bit prevents training altogether, resulting in the model failing to progress with any learning using E7M7, confirming previous findings [] that neural network training is more sensitive to exponent bits than mantissa bits. To analyze the cause, we first conduct an ablation on the clamping mechanism by either removing only the inner or outer exponent range, as depicted in Figure 4. We find that models with only their inner exponent ranges clamped train normally while models with only their outer exponent ranges clamped could not engage in training, indicating that the inability to represent large values is the cause of failure for E7M7.

We also found that specific regions of the model are more sensitive to reduced precision than others. For the Llama models, removing the masking from the LM head was sufficient to restore training, although the resulting model was much more unstable. This result supports the conclusion that certain parts of the model are more sensitive to reduced precision than others, as no training occurs when the entire model is trained with E7M7.

### 4.3 Loss landscape sharpness

To further uncover the relationship between bit width and training robustness, we use Equation 2 to quantify the degree of training instability increase by measuring the loss landscape instability of Llama 7B models in Table 1 and Llama 120M models in Figure 6.

In Figure 6, we show Llama 120M models trained until training loss divergence occurs, as well as plotting the loss landscape sharpness values of the models in E8M3, E8M4, and E8M5. Although the points of divergence are different for each of the models, we can see a general trend of increasing sharpness until the model diverges sharply, after which it cannot revert to its original training trajectory. Training required approximately 2 hours per 10,000 training steps on a node with eight A100 GPUs.

To verify that similar behavior occurs in larger models, we include a comparison of the training losses of Llama v2 models with 7B parameters trained for 5,000 steps in Figure 7. We apply early stopping at 5,000 training steps because training a Llama 7B model for 5K steps requires approximately one week on a node with eight A100 GPUs. From these experiments, we can see that loss divergence is visible in the E8M3 and E8M4 models, while it has yet to emerge in the E8M5 model. However, from Table 1, we can see that the loss landscape sharpness continues to increase for the E8M5 model, even though no signs of instability are yet visible.

The E8M3 and E8M4 models show much higher sharpness values and both accordingly diverge early in training. In contrast, there is only a gradual increase in the loss-landscape sharpness for the E8M7 runs. As seen in Figures 7 and 6, we observe that models gradually increase in sharpness until a threshold is reached, although the threshold value may differ depending on the configurations. These results suggest that models with fewer mantissa bits are entering regions of ever greater instability during training, even when these instabilities are not visible in the loss curve. We believe that, in the future, such analysis of loss landscape sharpness can be used to identify when the model is at risk of loss divergence.

### 4.4 Robustness to learning rate changes

We further attempt to identify hidden instability in E8M5, which did not diverge during the initial training stages in Figure 6. Inspired by Wortsman et al. [], we analyze the robustness of Llama 120M models to changes in the learning rate by comparing training at BF16 with that for E8M5. As seen in Figure 8, the E8M5 training runs have more frequent loss spikes during training, especially when the learning rate is increased to $4e\mathrm{-3}$. Although no cases of loss divergence were found, we believe that the higher frequency of loss spikes indicates greater instability during training, supporting our claim that the loss landscape is sharper for E8M5 even before loss divergence occurs.

Figure: Figure 8: Comparison between Llama 120M models trained using E8M5 masked training (left) and standard BF16 training (right) for $lr=4e\mathrm{-4}$ (the default learning rate) and $lr=4e\mathrm{-3}$. Using 18 random seeds per configuration, the E8M5 runs show more frequent loss spikes, especially at the higher learning rate, indicating greater training instability.
Refer to caption: extracted/5628109/images/tinyllama120m_lr_x10_e8m5.png

## 5 Discussion

This work aims to propose quantitative evaluations and analyses of training instabilities that occur when reducing floating-point precision. By clamping unrepresentable exponent values and masking out the mantissa before the computation, we can approximate matrix multiplication with reduced-precision computation and high-precision accumulation. Our experiments have shown that reducing the precision of floating-point computations destabilizes LLM training and that existing mechanisms to stabilize FP8 training do not offer sufficient robustness as to allow their cost-effective use.

Again, we emphasize that our work does not aim to disprove the viability of FP8 training. We merely argue that FP8 training, in its current form, is uneconomical, not impractical. Even if FP8 training is effective for carefully selected hyperparameters under specific conditions, our assertion is that the costs of finding such conditions and the risks involved in training a less stable model outweigh the benefits of using FP8 for accelerated computation. Instead, we propose methods to evaluate the stability and robustness of reduced-precision training, which is key for FP8 or other reduced-precision training schemes to be viable for LLM training in the real world.

Also, we would like to preempt misunderstanding by making clear that we are not questioning the usefulness of FP8 for inference. Several recent works [, , ] have shown that it is even possible to quantize LLM weights to below 4 bits without sacrificing much accuracy. Xia et al. [] has also shown that, even for the A100, using FP6 for inference is a viable option. We believe that dedicated FP8 and FP4 processors can make LLM inference simpler to implement and faster to compute.

We note that, from our experiments, several methods naturally suggest themselves as possible stabilization techniques. First, the initial stages of training could be conducted in higher precision, similar to how smaller batch sizes may be used during the initial stages of training []. Increasing the precision when the loss landscape becomes too sharp and returning to lower precision when the model is in a smoother region may also provide a tradeoff between training speed and stability.

Second, the layers that are more sensitive to reduced precision may be kept at high precision, while only the less sensitive model layers are computed with reduced precision. For example, during our experiments, we found that removing masking from the LM head of a Llama model was sufficient to enable E7M7 training, although the resulting model was less stable. For GPT models, we found that increasing the precision of the first two decoder blocks to TF32 was sufficient to prevent loss divergence. However, as such compensatory techniques depend on the model architecture, training data, and other aspects of the training environment, we leave their investigation to future work.

## 6 Limitations

A limitation of this work is that it focuses on the initial stages of pre-training when many instabilities are known to arise only later in training []. For example, Wortsman et al. [] show that the logits of the outputs diverge from zero only at the later stages of training. To this, we argue that our studies likely underestimate the instabilities that FP8 or other reduced precision training schemes will face, further strengthening our case that reduced-precision training methods are too unstable to be profitably utilized in their current form.

Second, despite finding that the exponent bits are of greater importance to LLM training than mantissa bits, we were unable to experiment by increasing the number of exponent bits. This was because matrix multiplication in FP64 is over an order of magnitude slower than BF16 on A100 and H100 GPUs when using tensor cores. Experiments using representations such as E11M4, created by removing 48 mantissa bits from FP64, may be illuminating, but we found it impractical to train models with a greater number of exponent bits.

Finally, our experiments are limited in that they only the training loss is used as an evaluation metric
instead of real-world natural language tasks such as MMLU [] scores. However, while lower perplexity is no guarantee of superior performance on downstream tasks, we believe that the divergence of the training loss is sufficient as an indicator of training failure.

## 7 Societal impact

As the costs of training or even fine-tuning LLMs continue to climb with the ever-growing size of the largest models and datasets, we believe that FP8 can be a useful tool in reducing the costs of LLMs to manageable levels. The recent introduction of FP8 in the NVIDIA Ada and Hopper [] series of GPUs and the even more recent announcement of FP4 support in the Blackwell [] series of GPUs is placing these low-precision data types within the reach of industry and academia, both of which groups are always in need of ever more compute.

Although we have focused on the difficulties of using FP8, with the electricity consumption of data centers projected to double within the next few years [], mostly due to increased artificial intelligence workloads, we believe that any improvement in the training efficiency of LLMs would have a noticeable impact on global electricity consumption. As such, it is our hope that future work uses our work as a guide to improving the ways in which low-precision computations can be used.

## 8 Conclusion

We demonstrate that the training stability of LLMs decreases incrementally with the reduction of floating-point bit widths used for training models of up to 7B parameters. Using our proposed loss landscape sharpness metric, we measure the gradual increase of instability that leads to loss divergence, shedding light on a phenomenon with potentially large financial and environmental costs.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: - In Figure 6, where MS-AMP was applied to nanoGPT with and without the LM head in FP8, what does the observed performance degradation indicate about the LM head’s role in stability? And how do the loss trends compare to BF16 training based on the exponential moving averages?

- Table 1 shows loss landscape sharpness values for Llama v2 7B models at different precisions (E8M3 to E8M7) over various training steps; how do these sharpness trends reveal the model's progress towards loss divergence, and are there any specific sharpness thresholds that clearly predict instability?

- The stability of E8M5 configurations is tested under different learning rates in Figure 9, showing more frequent loss spikes at higher rates—how does this pattern of sharpness changes affect hyperparameter tuning in low-precision training, and what insights do these learning rate sensitivities give for FP8 compared to BF16?
soundness: 2
strengths: 1. It’s got this new loss landscape sharpness metric that basically acts like an early-warning system for training issues, spotting instability before it even shows up in the loss curve which helps predict when training might go off-track.

2. Unlike some of the latest studies that stick to small-scale models, this paper actually dives into FP8 precision with full-size LLMs, so the findings are more practical and closer to real-world training needs.

3. They tried out reducing bits both on exponent and mantissa, making it unique since other papers usually just mess with mantissa bits, giving a fuller picture of how bit reduction affects training stability.

4. The paper shows in real-world training that FP8 doesn’t handle hyperparameters as well as BF16, which limits its ability to actually save money in training, something that hasn’t been looked into much in other studies.

5. By using TinyLlama for their experiments, this study connects the theory of low-bit precision with real stability checks, giving a strong framework for researchers wanting to make low-precision training more stable in LLMs.
summary: The authors explores what happens when large language models (LLMs) are trained using FP8 precision, meaning only 8 bits are used for floating-point numbers. FP8 is theoretically cheaper, but it introduces significant instability compared to BF16, the precision currently preferred for these models. The authors introduce a new metric to measure training stability by examining "loss landscape sharpness," which helps predict when training might diverge. They found that FP8 limits the range of hyperparameters that work, reducing its potential cost-savings. Overall, FP8 isn’t quite suitable yet for full-scale LLM training without further stabilization methods.
weaknesses: - The study mostly measures training stability by focusing on loss landscape sharpness and tracking divergence in training loss, which is useful but doesn’t fully reveal how well the model would perform on real-world tasks like NLP benchmarks. This narrow focus on training metrics rather than practical testing limits the broader application of the findings in real scenarios.

- They mention computational limits for testing with exponent bits, so they mainly tried reducing bits without exploring the impact of increasing them. Some recent research suggests adding exponent bits could help stabilize FP8, leaving an area of potential solutions unexplored.

- Because running full-scale experiments is costly, they used smaller models like TinyLlama to approximate larger ones. But recent studies point out that these smaller models don’t always behave the same as big ones like GPT-3, so the results might not directly translate to large-scale models.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: - Could you maybe compare your proposed metric to the metric proposed in Keskar et a. (2017) or other SOTA metrics in terms of computational costs and early detection of training instabilities? 

- Could you maybe run the experiments on more training schedules (i.e., initial learning rate, learning rate) than the ones mentioned in the first paragraph of the Methods section to verify the consistency of your observed results? 

- Could you maybe include the results (i.e., a table or plot) showing the failure of training with lower exponent bits (e.g. E7E7)?

- Could you maybe verify the two suggestions (i.e., high-precision initial training stages and higher precision for sensitive layers and vice versa for less sensitive layers) proposed in the last paragraph of the "Discussion" section as possible training stabilization techniques?
soundness: 2
strengths: The proposed metric is an effective, computationally cheap, proxy measure of the underlying training instability. The paper clearly shows in Figure 8 and Table 1 that the proposed metric can detect the training divergence before it occurs for the E8M5 and compare the results with other reduced floating-point precision (for example) 

The paper's outcomes confirm previous findings that neural network training is more sensitive to exponent bit reductions than to mantissa bits.
summary: The paper proposes a new metric to evaluate the training instabilities of LLMs when reducing the floating precision of the model's weights. The paper introduces a novel loss landscape metric based on the previously proposed Keskar et al. (2017). However, the paper's novel metric is computed on the last token's logits rather than on the model's input for the computation. Thanks to this change, the computational cost of computing the metric is significantly reduced since the metric has to be computed only once for each measurement. This choice is motivated by the autoregressive decoder-only model's last token, which receives inputs from all other tokens. 

By proposing this novel metric, the paper investigates the cost-effectiveness and impact of reducing the model's floating point precision. The paper analyzes the impact of reducing the model's weight precision regarding training instabilities by conducting an ablation study. They conducted experiments by simulating incremental bit reduction of the mantissa or exponent bits of the model's weights.
weaknesses: The paper proposes a novel and computationally cheap metric to detect training divergence based on the work of Keskar et al. (2017). However, the paper does not compare the performances or computational costs of previously proposed methods in the literature. Also, the paper does not consider or compare methods exploiting the gradient or the internal activations of the model to detect training divergences.  

The paper does not consider different training hyperparameters from those reported at the end of the first paragraph of the "Method" section.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: -Many previous works have proposed techniques to address numerical instability, such as precision decoupling and automatic scaling in MS-AMP. Were these techniques or similar techniques applied in your experiments, particularly in the simulations conducted in sections 4.2 and 4.3?

-I'm unsure about the data quantity and quality used in this work, as the experimental settings aren't clearly described. For instance, it's unclear what dataset was used in Figure 6, and the training steps don't intuitively reflect how many tokens were trained.
soundness: 2
strengths: -The motivation of this paper is compelling, particularly the concern regarding the stability of FP8 compared to FP16 and BF16. This raises important questions about the cost-effectiveness of FP8 for LLM training. A deep analysis and understanding of the effects of low-bit training can significantly benefit the research community by providing valuable insights.
-The introduction of a new metric for quantifying loss landscape sharpness is good. This analytical framework could be beneficial for the research community.
summary: This paper explores the feasibility of using reduced-precision floating-point representations, specifically FP8, for large language model (LLM) training to enhance computational efficiency. While BF16 has become standard due to its balance of precision and performance, FP8 poses challenges in stability. The authors propose a new metric to quantify loss landscape sharpness, aiding in predicting training stability. They analyze the effects of reducing mantissa and exponent bits, finding that exponent bit reduction significantly impacts model performance. The contributions include a novel approach to evaluating training stability and insights into the trade-offs of using lower precision in LLM training.
weaknesses: My primary concern with this paper is the confidence in the experimental validation, especially given the smaller scale of the models used. Theoretically, we understand that reduced precision can affect model convergence. However, my intuition suggests that as models grow larger and datasets become more extensive and of higher quality, the models' robustness improves, potentially mitigating the impact of low-bit precision. This phenomenon is also observed in low-bit post-training quantization, where larger models are less affected by quantization. I am uncertain if the observed effects would persist in larger models. 

As this paper is based on MS-AMP. The MS-AMP paper, which tested with 7B, 13B, and 175B models on more diverse datasets, concluded that FP8 training loss is consistent with FP16. This seems contradictory to the findings of this paper. Is there a difference in experimental settings, or did I miss something?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1) Please add comparison to previous sharpness metric, showing why the proposed method is more suitable for autoregressive models.
2) Does the bit reduction experiments include scaling when reducing the exponent bits. If not, I suggest to add it- I believe in that cases we will see completely different conclusions.
3) How do you implement the experiment of reducing mantissa bits? For the exponent it is well explained in the paper (clipping). Moreover, the author claim their method of masking is an imperfect approximation - can you please explain what is the gap? How do you show this gap is not critical.

4) Please try to clarify why the authors are not able to converge FP8 training while 2 previous papers are able to converge.


===================================================================

Rebuttal:

Dear Authors,

Thank you for taking the time to address the questions and concerns raised.

Unfortunately, I believe the primary issue remains unresolved. The authors attribute the divergence in their FP8 training to the use of an open-source dataset. However, Intel’s paper demonstrates successful FP8 convergence using open-source datasets as well. Additionally, while the authors suggest that Intel’s findings support their claims, I believe this is not accurate. In Intel’s work, the divergence is attributed to the use of the SwiGLU activation function, which is not employed in this paper (as nanoGPT uses the GeLU activation function).

Furthermore, I believe that the absence of scaling and the bias introduced by the bitmask in the mantissa (which could be critical during the backward pass) may significantly impact the conclusions drawn in this paper.

I will maintain my score.
soundness: 2
strengths: The paper suggest a sharpness metric to help to check where a model training will diverge  The the authors claims that the proposed method is more suitable for autoregressive models. I think that comparison to previous method, showing cases where the proposed method works better than previous one in autoregressive models is a must.
summary: The paper investigates training using low precision, specifically the FP8 datatype. It examines the stability of FP8 training across various seeds and initializations, revealing loss spikes during certain training phases. To further understand these issues, the authors analyze training stability by simulating different numbers of mantissa bits. Finally, they propose a method to assess loss sharpness and predict potential training divergence.
weaknesses: I think the weaker part of the paper is the experiment section, mainly  the main MS-AMP experiment (section 4.1). - which is the main motivations for all the paper.
The authors claims that in all cases FP8 training is not able to get the same results as the baseline (BF16) counterpart. We can find 2 papers - from Microsoft (https://arxiv.org/pdf/2310.18313) and Intel (https://arxiv.org/pdf/2409.12517) showing that is is possible to train LLM in FP8 - I suggest the authors to investigate the gap between their paper and these.

Meta-review: The paper addresses the important problem of training LLMs in reduced-precision floating-point representations. The scope is to find if the available methods for training in FP8 precision are robust enough. In this context, the paper proposes new evaluation strategies for the sharpness of the LLM training loss.

Overall, the paper has merits, but the reviews are mixed about the quality of the work. In particular, the impact of the trained models on real-world NLP benchmarks is not exhaustively assessed. Furthermore, the authors were encouraged to compare to SOTA FT8 training methods. On the other hand, the reviewers pointed out that experiments with larger model sizes are necessary to fully evaluate the impact of the FP8 training. Last but not least, it is not clear whether the divergence in the FP8 training is due to the use of an open-source dataset, which is not supported by prior work (reviewer nxKU).

While I do see a potential impact of this paper on the community, I remain of the opinion that the paper is currently not mature enough for acceptance.

Decision: Reject