Towards Fully FP8 GEMM LLM Training at Scale
Authors: Alejandro Hernández-Cano
Dhia Garbaya
Imanol Schlag
Martin Jaggi
Venue: NeurIPS 2025 poster ()
Keywords: FP8
LLMs
Training Dynamics
Quantization
Efficient Training
Primary area: deep_learning
TL;DR: We perform an important step towards LLM pure FP8 training by enabling stable FP8 dot product attention reaching new throughput records

Abstract: Despite the significant potential of FP8 data formats for large language model (LLM) pre-training, their adoption has been limited due to challenges in maintaining stability at scale. Existing approaches often rely on suboptimal fine-grained FP8 kernels or fall back to higher-precision matrix multiplications (GEMMs) in sensitive components, such as attention projections, compromising potential throughput gains.
We introduce a new class of LLM architectures that, for the first time, support FP8 computation for all GEMMs within transformer blocks during both forward and backward passes. This enables unprecedented throughput gains, particularly at scale, while matching the downstream performance of standard BF16 training. Our architecture design reduces large outlier activations, promoting stable long-term FP8 training. Additionally, we identify key metrics for monitoring low-precision training and predicting potential future divergences.

Towards Fully FP8 GEMM LLM Training at Scale
Alejandro Hernández-Cano"
Dhia Garbaya' EPFL
EPFL
alejandro. hermandezzaano@pff..ch
dhia. garbaya@epfl.cl ch
Imanol Schlag
Martin Jaggi EPFL jartin.jaggi@epfl....
ETHZ
ischlag@ethz.cl ch
Abstract
Despite the significant potential of FP8 data formats for large language model (LLM)) pre-training, their adoption has been limited due challenges in maintain- ing stability at scale. Existing approaches often rely suboptimal fine-grained FP8 kernels or fall back to higher-precissee in sensitive components, such as attention projections, compromising potential throughput gains. We introduce a new class of LLM architectures that, for the first time, support FP8 computation for all GEMMs within transformer blocks during both forward and backward passes. This enables unprecedented throughput gains, particularly scale, while matching the downstream performance of standard BF16 training. Our architecture design reduces large outlier activations, promoting stable long-term FP8 training. In addition, identify key metries to monitor low-precision training and predict potential future divergences.
matrix multiplications (GEMMs)
Introduction
Recent progress in the training of transformer-bused Large Language Models (LLMs) has significantly advanced the field of language modelling. Scaling up both model size and training data remains a reliable strategy to enhance their performance [15] Consequently, state-of-the-an models are requiring substantial computational
typically trained scale using extensive datasets 1
25
resources-ofee in the order of millions of GPU hours.
Thus, the development of efficient training techniques has become increasingly essential. One of the main research avenues for efficiency is the use of lower-precision number formats to accelerate training on appropriate hardware accelerators. Recently, the of 8-bit floating-point (FP8) formats has shown promising results 3 6 24]. However, the widespread adoption of current approaches still limited due to suboptimal throughput benefits, One cause of slowdowns the use of higher precision in those General Matrix Multiplications (GEMMs) which are most sensitive, such as attention score computation, while another issue is the overhead caused by more granular FP8 scaling strategies. One of the key challenges in 8-bit LLM training originates from the relatively narrow dynamic range offered by FP8 formats and thus higher risk of underflows and overflows, especially with the prevalence of large outlier features observed in the LLM's 'ss neural activations effect of outliers on quantization later in Appendix C To mitigate this issue, modern FP8 training recipes utilise various scaling techniques before casting from higher-precision
during training 30, [. We formalize
crrmats-ttpicalle BF16 [14] activations to FPS formats used in matrix multiplications. These scaling approaches help maximize the effective use of FP8's limited dynamic range, reducing the risk of underflows and overflows.
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent work has introduced promising FP8 training recipes by employing multiple scaling factors per single tensor [3, allowing for finer and more precise casting to lower precision. Yet, this comes with an efficiency overhead, diminishing the large gains initially expected from using FP8. Another strategy involves adjusting the standard SwiGLUU-based transformer architecture (28] to prevent emergent outliers from occurring [6]. This area of optimization remains underexplored, as most works focus on FP8 GEMMs within the linear projections within the transformer, while maintaining higher precision for other GEMMs, namely those involved in the dot product attention mechanism. We refer to such training strategies simply as FP8 training. We label the approach of also including FP8 attention computation as FPSDPA.
the Fast and designed to mitigate large activation outliers and enable efficient large-scale FP8 training with low- overhead scaling strategies. For the first time to our knowledge, this approach enables FP8 GEMMs not only in the linear projection, but also within the attention mechanism of each transformer block, achieving unprecedented throughput improvements of up to 43% in the 8B parameter model scale, while maintaining equivalent downstream performance compared to higher precision baselines. In addition, we present a comprchensive recipe for monitoring, explaining, and predicting training instabilities that might not surface in the early stages of training. This approach provides researchers with greater confidence in the long-term stability of FP8 training recipes, reducing the need for costly, full-scale experiments when using new architectures. Furthermore, provide an interestingly useful observation about larger models' tendency to diverge later in training with FP8.
In this paper, we introduce FOG:
set of LLM architectures specifically
Our key contributions are the following:
We introduce the FOG set of architectures, designed minimise outlier features during training. Our recipe allows stable training with FP8 computation of all GEMMs inside the transformer blocks, surpassing the throughput of the standard BF16 approach by up to 43%.
Our design achieves equivalent quality results to BF16 baselines, while providing signifi- cant speed-up. We empirically attest both performance and stability on various model sizes (0.4B, 1.5B, 8B) and data regimes up to 15x the Chinchilla optimal data budget [12].
We show the flexibility of FOG design as it can be adapted to several architectures including various families of activation functions and even
ofatture-of-xpperrr (MoE) settings.
Using kurtosis, we provide recipe to judge an architecture's robustness to FP8 training in long data regimes using diagnostics from shorter runs We use this recipe to explain previously observed divergence behaviour at scale, and offer wide range of empirical results to demonstrate its usefulness. We believe this contribution allows FP8 training insights future transformer variants developed by the community, without the need for expensive full-scale experiments.
Background
Due to its limited dynamic range, FP8 tensors are particularly prone to overllows and underflows when representing extreme values. The FP8 formats come in two standard forms (20]: E4M3 and E5M2, each with different trade-offs. The first format, with four exponent bits and three mantissa bits, offers higher precision. In contrast, the E5M2 format, with five exponent bits and two mantissa bits, provides a broader dynamic range at the cost of reduced precision. Existing large-scale distributed training frameworks such as DeepSpeed or Megatron 229 leverage this distinction by employing E.M3 for tensors in the forward pass to maintain precision and E5M2 for the backward pass to handle the broader dynamic range of gradients effectively. Nonetheless, both formats have much lower representation capacity than half- or are applied when casting tensors down to FP8 in order to make more efficient use of this restricted range. These strategies are mainly tensorwise and fall into two main categories: delayed scaling and just-in-time scaling (JIT). Delayed scaling uses information from previous training iterations to determine the scaling factor of the tensor for the ongoing iteration, requiring single pass on the data along with storing a short history of useful metrics observed across an interval of past iterations. JIT scaling, on the other hand, can hinder the gains from using FP8 because it uses the distribution of the tensor being produced in higher precision-to compute the scalar, before casting the input and performing the GEMM in FP8, requiring at least two passes through data. A more recent approach aims to make scaling more robust by using multiple scaling factors per tensor, allowing different
formats. Therefore, various scaling strategies
2

---

tensor blocks to have different scaling factors [2 This leads to more precise FP8 casting within each block. Naturally, this finer scaling strategy induces a larger overhead on such GEMM kernels relative to the tensorwise delayed scaling recipe.
Ensuring stable FP8 training remains challenging. It becomes problematic when certain activations produce large outliers during training, making such low-precision representation unfeasible and leading rapid divergence. Prior work introduced the term massive activations, phenomenon similar to outlier features, and showed their crucial role in LLMs' capabilities [30]. Understanding the dynamics of these outliers is crucial for explaining FP8 divergence and identifying the network components responsible for them. One notable source of such outliers' amplification has been identified to be the widely adopted SwiGLU (Swish Lenter- Unit) activation function [28]. Replacing it with scaled variant, SmoothSwiGLU regulates large outliers and was shown to stabilize previously diverging FP8 training runs and ensure their convergence (6]
Further examination has shown that not only is SwiGLU an outlier amplifier, but Gated Linear Units (GLUs) in general, as well as pre-normalization layers, suggesting that improper signal propagation is the root cause of outliers [9]. Removing these components and equipping transformers with QK entropy regularization mechanisms such as QK RMS Normalization [TO, producing the Outlier Protected (OP) architectures [9], has been shown to diminish late-stage outliers observed by orders of magnitude. while providing equivalent prediction quality. While OP architectures were shown to be beneficial for post-training quantization, its use for FP8 pre-training remains unexplored. Finally, an alternative pre-normalization layers are post-normalization layers [8] Long data regime trainings have confirmed their superiority in terms of training stability with the standard BF16 mixed precision training (1
3 FOG: Fast and Outlier-Guarderr FP8-suited architectures
Our architecture base, as illustrated in Figure T. makes key changes to widely-used transformer networks [32]. The pre-normalization block before the attention mechanism and FFN is removed. In addi- tion, normalization mechanism in the attention is added prevent entropy collapse, key training instability in transformers (34), from occurring. This mechanism can take the form of a QK RMSNorm block 110:
FOG
Rorralaation
Urear
||[|| VD
(x) rmns(x) Y x
rmn(x):
where x E RD, Y E RN is the learnable gains vector, @ the Hadamard product and || la is the fz-norm. Alternatively, the (x) = tanh(axa) element-wise activation function, where E IR is trainable, can be applied to query and key tensors. This activation has been shown to have regularization effects akin to RMS normalization blocks (35], while being computationally more efficient.
?
Core Streetoe
Further, the input of the first transformer block is scaled by a-1 to maintain unit variance activations initialization, where is the chosen standard deviation of the network's 's random initializa- tion. Finally, to enhance performance, a learnable normalization block is applied before the residual connections. This takes the form of a LayerScale 31 block, LayerScale, (x) := 7 @ x, where 7 E RDD is a Icarnable gain vector, or an RMSNorm block, result- ing in normanorda architecture [8] In both cases, the learnable gains vector is initialized to 1/ /num Jayers and keeps the residual branch unnormalized, allowing proper signal propagation (9] Our architecture suite is specified in Table D and further details are available in Appendix B
Figure 1: FOG transformer.
While the OP architecture already offers several guards prevent large outliers from occurring, we observed that it remains an impractical choice for FP8DPA training. In Section[3.]] [e we show that, like all other architectures tested, it suffers a fatal loss divergence early during training. We isolate the two components responsible for OP's incompatibility with FP8DPA training: the trainable QK RMSNorm gains vector 7, and the lack of any normalization. We identify the use of post-normalizzaioo as not prone to the outlier tendency pre- normaaaaaiiationn networks have.
3

---

Model
QK- Re-Regularization
Activation
Normalization
FOG-max
RMSNorm* RMSNorm*
x|ELU
13
Dost-MMSNrm
FOG-opt
GeLU
Post-RMSNorm
FOG-flash
Tanh*
GeLU
Post-RMSNorm
OP 9
RMSNorm
GeLU
LayerScale
Table 1: FOG architecture suite compared with OP. Regularizations marked with * indicate that gains are not trainable. Each variant offers different trade-offs, with FOG-flash having the higher throughput and FOG-max observed have better downstream quality.
FOG-opt
4.35
1.13
3.70
1.25
Crodlennt
3.35
0.75
2.70
0.25
30 Conounal Tokert
Cansonal Tileet
Figure 2: From OP to FOG-opt step by step. Comparison of 390M models under FP8DPA training. The first architecture to diverge is OP, while OP with frozen QK RMSNorm gains survives the stable phase of training. still, however, experiences significant divergence during the learning rate cooldown, which starts around 42B tokens in. The architecture that converges, FOG-opt, is the result of adding post- nosttaormalizatinn to the previous recipe. Gradient norm reported is the 200-rolling- window mean and 5%-95% quantile bands.
Figure 2 ablates the components transitioning from OP to FOG-opt. We can see that freezing the trainable QK RMSNNrm gains results in significantly more stable training. We attribute the early divergence of OP to the fact that uncontrolled QK normalization leads to an explosion of its gains when training in low precision. Note that these gains generally not weight-decayed. We experimentally observe this explosion, confirm that using L2 regularization helps delay the divergence. We finally opted for freezing the gains to constant value as it is simpler and sufficient, doesn't t compromise performance, and offers small speedup. Our ablations highlight that constant value for the gains slightly greater than improves loss. Therefore, to retain its benefit after removing the y gains vector, increase the standard 1/ /D/t attention softmax scale tiny optimization trick offering equivalent attention score matrix S:
Nr. (Q), (K) 1
2o Q rims Q) )
KoKK rmss(K)
T
S = v D
(K)
DDyh
D ak
Finally, we empirically show that the addition post-normalization is important to ensure conver- gence with FPSDPA during the learning rate decay phase.
Prior works also favored post-normalization over pre-normalization better stability in BF16 training. We extend this observation to our FP8 setting and we confirm that learnable LayerScale blocks alone, even with controlled QK regularization, cannot ensure convergence during this last phase. We attribute this late divergence of OP to the fact that LayerScale blocks without normalizations are not enough handle FP8 outliers, potentially due the considerable changes in model statistics following the learning rate decay, that are summed up in the residual connections resulting in huge activation outliers pattern of outliers on each transformer block's 's output in Figure We note that we initially tested the idea of cooling down the previously constant weight decay during the learning cooldown phase, aiming to conserve model weights' norm [16] The Appendix shows that such intervention has no noticeable effect on stability neither performance. We chose to conserve this decision for all FOG runs for consistency and fair comparison across ablations.
providing evidence of their
last layers, as highlighted by the increasing

---

Layer
10
QXV
Consanuel TTolens
Figure 3: Kurtosis of QKV tensors during FPSDPA learning rate cooldown with OP+frozenQK architecture. Later layers exhibit significantly larger activation outliers.
4 Long-term outlier dynamics
To analyse the outliers present neural network activations, we use kurtosis as metric of the extremity of deviations of activation values (such as by outliers). We define the kurtosis kurt(x) of vector x E IR. as the scalar
kurt(x) := m((x1]
x² xx
where j and x² are the sample mean and variance, respectively, and exponentiation is taken element- w
wise. Given an activation tensor X E R NNxOX where N, C, and D are the batch size, se- quence length, and hidden size respectively, we define its kurtosis as the average kurt(X) NC Σ> ΣS kurt(Xne).
Under this definition, kurt(x) is maximized when few elements of x reach extremely large values, relative to the variance across the entire vector, i.e., when large outlier features are present. This definition has been used analyse outliers in BF16 training in previous work [9] and, unlike the standard definition kurtosis [22) in the probability theory literature, this definition does not center have zero-mean. For use. this is consistent with the fact that FP8 kernels do not shift their inputs before scaling and casting down. We track the dynamics of kurtosis in key activations. Namely, the inputs of the second projection in FFNs, the QKV matrix, and the output of each transformer block. Unless explicitly stated, we report the average activation kurtosis across all layers.
Using these activations, during training. Figure/ddemonssrratee an equivalent loss progression to the baseline while offering up to orders of magnitude lower kurtosis in some activations. Note that, unlike previous FP8 approaches, FOG architectures are trained with FP8 attention computations, introducing more quantization errors. As result, the kurtosis of key, query, and value projections becomes particularly relevant.
analyse the emergence of large outlier features at different stages
Activation Functions Baseline Llama exhibits late divergence during FP8 training (with attention in BF16), which has been attributed solely to the quadratic behavior of its gated activation func- eien-nng when weights become sufficiently aligned late training [6]. In our extended 450B token run using the FOG-max architecture, we employ the inherently quadratic xIELU activation function, see Equation (2). and observe stable training with kurtosis levels orders of magnitude lower than those of baseline Llama. In fact, modifying the FOG-max architecture to use the SwiGLU activation function resulted in stable FPSDPA training behaviour, as disscussed in Appendix D These results strongly suggests that architectures biased towards low kurtosis activations during training enable the stable use of quadratic activations, and challenges the completeness of prior explanations. This is particularly interesting given that such activations known produce linear gradients, which benefit the backward pass-likely contributing to FOG-max's superior performance over GeLU-based variants as seen in Section
Long-term outlier growth These architectures exhibit sub-linear to logarithmic trend in the long-term growth of QKV outliers, consistently shown by kurtosis in Figure 4 This behavior
5

---

POG-max
14
12
2.8
QKV Kurseis
10P
Transformor Kurooiis
PTN Kartoss
201
20%
10"
20
40
60
80
100
20
40
60
80
100
Consuned Tokoms
X10'
Cousurmed
x1PP
Figure 4: Loss and kurtosis training dynamics of 1.5B FOG-max and Llama3 models trained for over 100B tokens with BF16 precision. Loss reported is the 200-rolling- winddowwwwwwwwwww-o mean and 5%-95% quantile bands.
supports their robustness to FPSDPA, it suggests that prohibitively longer training would be required to see a substantial increase kurtosis. Our extended run is consistent with the hypothesis it does not exhibit any sign of divergence.
Kurtosis early signal Figure 5 shows example of a diverging FP8DPA run, comparing it with the successful FOG-max training. This emphasises the importance of tracking tensor-level metrics such as kurtosis to potentially predict later divergences, before common global metrics like the loss and gradient norms show any symptoms of divergence. this example, while loss irrefutably diverged around the 15B token mark and the gradient norm consistently spiked no earlier than 12B tokens, the QKV kurtosis was already diverging from the expected sub-linear growth consistently seen across different architectures as early the 3B mark, giving a potential early divergence sign.
DLAS6
170
1.5
1.0
20'
2.70
2.00
10
10
10
Doseined Thbens
Ccoenned Tikeen
Thiees
Figure 5: Training dynamics of failed and successful FP8DPA run. Kurtosis exhibits atypical behaviour much earlier than when the loss diverged. Gradient norm reported is the 200-rolling- window mean and 5%-95% quantile bands.
5 Experimental Results
We perform extensive experiments to verify our architecture across several scales. We use the
Fed (33 text corpus, filtering out any web opt-out domains with robots. txt, resulting in a rigorous data-compliant corpus (5). The data is tokenised using 131K vocabulary BPE

---

tokenizer. We keep consistent context length of 4096 during all main experiments. In terms of the optimisation algorithm, we use AdamW W [9 with default hyperparameters. Our learning rate schedule is comprised of three phases: Warm-up, Steady, and Decay phases (WSD), as it has been shown to provide equivalent performance to the cosine schedule []. while allowing to train beyond fixed training durations. For the models, we train 390M, 1.5B and 8B parameter models for different token counts, specified at each experiment. Our baseline architecture follows the Llama3 8B model design 77 with the 390M and 1.5B being adapted to their respective sizes. Since Llama3 uses gated linear unit, unlike the OP and FOG variants, we increase the FFN sizes of OP and FOG to maintain an equal parameter count. Further details regarding architectures and hyperparameters available in Appendices A and
Our ware infrastructure consists of nodes with 4 Nvidia Grace Hopper GPUs each. Our dis- tributed training framework adapted from Megatron-L.M (29), which uses Transformer Engine FP8 recipes. With 390M parameters, our experiments reach 50B tokens. We scaled 1.5B experiments 125B tokens to obtain more meaningful evaluations. In addition to the absence of late-in-training outlier amplification from FOG's non-gated activation functions and our kurtosis progression guar- antees, we further validate our method's s stability on long data regimes by continuing pretraining FOG-max up to 450B tokens. Finally, we scale the model size to 8B and train for 20B tokens. We show the divergence of other architectures with FP&DPA while FOG variants converge and match the baseline Llama3 BF16 loss, while being 35%43% faster. During all experiments, delayed scaling strategy, with a margin of zero and a history length of 1024 steps.
the FP8
We make our implementation, along with detailed steps for our experiments, public under the repository https::/
com/ http://w.....oooooooonsneeeeeeee ccmm///noonyoous44555/////
5.1 FP8 stability
We compare our approach with different architectures proposed in the literature. Namely, the OP architecture, OLMo2, Llama3, and Llama3 with the SmoothSwiGLU activation following the previous work [6], adapting each network to 390M and 1.5B parameter count. In the case of the Llama3 baseline, we also provide results on the 8B scale. Results are shown in Figure 6 This experiment displays the unsuitability of existing architectures for FP8DPA training, as all of them diverge. For the case the OP and OLMo2 architecture, despite having an attention miitigaatnn strategy the QK RMSNorm divergence is still observed, as discussed in Section 3
Liaend Seaat
2001 Seal
14B Sol:
53 Saae
Cocasual Takeo
x122
Oanamnel Tikana
XCUP
Oaoazzd Tiken
-10
Figure 6: Cross-entropy loss plots of different architectures with FP8DPA training. No other tested architecture was able to surpass the 20B token mark without diverging at any scale.
Another interesting observation from these experiments the tendency of larger models to diverge in later stages of training compared with similar but smaller models. We validate its consistency across architectures, as presented Table This observation not been raised before, possibly due the longer time needed for FP8 settings (with BF16 attention) to diverge, in contrast to FP8DPA training. While this trend could have many practical implications, exploring it fully falls outside the scope of this work, and encourage future research in this direction.

---

Divergence Mark
Architecture
Model Size
(in billions of tokens)
Llama3
390M
0.7
Llama3 1.5B
1.1
Llama3
8B
6.6
OLMo2
390M
3.3
OLMo2
1.5B
15,9
Table 2: Token mark when loss was observed to diverge.
5.2 Efficiency
Standard context length Table 3 explores the efficiency of FOG 1.5B and 8B model scales under FP8DPA training, using a standard context length of 4096. We compare our set-up with the BFI6 baseline and the stable Llama FP8 training with SmoothSwiGLU, which is, to the best of our knowledge, the only dense architecture proposal demonstrated to work at scale with FP8. Note that the AmoothSSiGGLU cannot benefit from enabling FP8 GEMMs in the attention mechanism, it was shown to suffer a big loss divergence Figure Note the increase in throughput gains as the model size increases. The GEMM input tensors increase in size and consume significantly more time during the overall weward-backward pass, compared with other operators.
Throughput
Size
Model
Precision
(tokens/sccond/GP))
Llama
BF16
9105
Jamma-SmoottSwiii..
FP8
12228 (+34.3%)
8B
FOG-max
FP8DPA
12344 (+35.5%)
FOG-opt
FP8DPA
12414 (+36.3%)
FOG-flash
FP8DPA
12764 (+40.2%)
Llama
BF16
46470
FOG-max
FPSDPA
53551 (+15.2%)
1.5B
FOG-opt
FPSDPA
53877
(+15.9%)
FOG-flash
FPSDPA
54848 18.0%)
Llama--SmotthSwiGLL
FP8
54903 (+18.1
Table 3: Training throughput measures with FOG versus other baselines. Using eight GH200 nodes with Zero-I sharding (26 for 8B models and single GH200 node for 1.5B models. Notably, in the 8B scale, all FOG variants outperform other architectures.
Long-context scenario Enabling stable training under FPSDPA regime leads to great benefits in long context scenarios, because the throughput becomes bottlenecked by the dot product attention computation with quadratic complexity. As result, we observe larger speed-up gaps between FOG and prior FP8 approaches sequence length increases, as demonstrated Table 4
Context
FOG-flash
llama+SmoothSwiiL..
Speed-up Gap
4096 (TP=1)
+42.6%
+38.5%
+4.1% 1
8192 (TP=1)
+43.5%
Out Of Memory
8192 (TP=2)
+39.1%
+34.2%
+4.9%
16384 (TP=2)
+38.8%
+3 1.1%%
+7.7
Table 4: Training throughput gains under varying sequence lengths (relative to Llama BF16), performed at 8B scale using GH200 nodes with a global batch size of 1024. Increasing to longer contexts required enabling Tensor Parallelism (TP). Raw throughput values reported in Table [I]

---

5.3 Downstream performance
We compare our proposals with the higher-precision Llama3 baseline across wide range of standard benchmarks to measure their downstream performance. Inference during down-stream evaluation uses BF16 precision. In T across a larger set of tasks, detailed in the Appendix G All FOG variants offer comparable down- stream performance with the higher precision Llama3 baseline with FOG-max architecture, even outperforming it. The 1.5B models are trained on 125B tokens, whereas smaller models are trained on 50B tokens.
report some of the most relevant scores along with average
Model
Hellaswag
ARC
PIQA
Average*
Llama 390M
33.5
47.9
65.0
39.8
FOG-max
36.5
36.3
62.9
62.5
68.0
68.2
41.2
40.8
FOG-opt
36.1
35.6
61.5
61.3
68.0
67.8
40.9
40.4
FOG-flash
35.9
35.2
61.5
60.4
68.1
68.2
40.5
40.3
Ilama 1.5B
43.7
71.8
72.5
46.1
FOG-max
43.3
43.4
71.6
73.0
72.6
73.3
46.0
47.1
FOG-opt
43.3
42.7
71.3
70.8
72.6
72.0
45.7
46.0
FOG-flash
42.8
41.9
70.9
69.4
72.2
72.0
45.7
44.9
Table 5: Performance across various fasks. For each task and model size, the first score results from the BF16 ablation and the second from the FP8DPA one. The average* is across larger set of tasks, show in Appendix G
5.4 Long-data regimes
To further justify the viability of FP8DPA long training with FOG, we train a 1.5B FOG-max on 450B tokens, way beyond the previously identified 200B tokens divergence mark of Llama2-7B (6. Note that our observation of smaller models' tendancy to diverge earlier with FP8DPA, and the long-term outlier analysis in Section further underline the sufficiency such a training duration.
We also switch to use FP16 optimizer states and BF16 gradients after 130B tokens, saving up memory previously used by full precision states, gradients, and model parameters master copy. We display the learning dynamics of our approach in Figure / The language modeling loss exhibits equivalent to better smoothness compared to the corresponding Llama baseline.
70G-mac (FPSDPA)
Llanaš (RP
27
2.
100
230
300
400
Consumed Tobens
X100
Figure 7: Long-data training regimes. FOG-max I 1.5 FP8DPA is trained on 450B tokens. The higher precision Llama3 experiment is included as reference. Note that a learning rate cooldown performed during the last 25B tokens of each experiment, following the WSD schedule. Loss reported is the 100-rolling-windee mean and 5%-95% quantile bands.

---

5.5 Additional results
SwiGLU This work mainly studies three architectural variants, sharing in common the use of point-wise activation functions. As mentioned in Section 4 we extended FOG to gated activation functions by showing its stability when using a gated MLP (SwiGLU), Details in Appendix D
FP8 optimizer moments We ran an additional FP8DPA experiment using FOG-flash following the same 390M scale setup but with bit optimizer moments. The loss converged smoothly to value of 2.645, nearly identical to the value 2.649 obtained with available in Appendix
moments, More details are
MoE We tested FOG under a Mixture-of-Eree -efee----tstt-teee (MoE) setting and the training was consistently stable. This experiment further supports the robustness of FOG. More details can be found in Appendix D
Limitations
Despite its robustness, record throughput boosts, and flexibility, the FOG set of architectures remains bounded by the following limitation. The final projection (LM head) is still performed in BF16. This operator is known to be very sensitive to outliers and has been used with half caff-precision in kawaaddbackwaadd FP8 training approaches, including ours. Due to computational constraints, decided to keep the study of this limitation for future work.
7 Conclusion
In this paper we demonstrate, for the first time, stable LLM training with fully FP8 matrix mul- tiplications within the transformer Mlocks-including the attention mechanism wechanism-without without sacrificing performance. We tested FP8DPA training across wide of previously proposed architectures and show that they consistently diverge early during training, highlighting the difficulty of FP8DPA training and novelty in our results. Moreover, contrast with other granular scaling recipes, we use the low-overhead delayed scaling FP8 strategy. Our design provides on-par downstream quality with the higher precision baseline, while offering up 1.5B model to 450B tokens, 15x the Chinchilla-optimal data budget for its size. Our work brings the community one step closer to fully FP8 GEMM training at scale i.e including the language modeling head. We further justify the long-term stability of our architecture by observing the outlier training dynamics across key activations by using kurtosis. The use of kurtosis to track outliers present during training was shown to provide meaningful insights to favour certain architectural components to predict future instabilities, as is a quantitative metric that measures outliers.
faster training 8B scale. We scale our
10

---

Acknowledgement
This work was initiated during the master's thesis of Alejandro Hernández Cano at EPFL. was supported as part of the Swiss AI Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a06 on Alps.
References
[1] Tom Brown, et al. Language Models Are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 34, pages 1877-1901, 2020.
[2] NVIDIA Corporation. Transf ormeréginin
NVIDIA/TTrantoomerEEgnine /Rthttps////tthubbccm//
[3] DeepSeek-AI, et al. DeepSeek-Ve -3 Technical Report. arXiv preprint, 2025. 01.4855//rxw..2412222
doi
[4] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Int3.Int80:: 8-Bit Matrix Multiplication for Transformers at Scale, In Advances in Neural Information Processing Systems, volume 36, pages 30318-30332, 2022.
[5] Dongyang Fan, et al. Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs. In Second Conference Language Modeling. 2025.
[6] Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling FP8 Training to Trillion-Token LLMs. In International Conference Learning Representations, volume 2025, pages 98631-98644 2025.
[7] Aaron Grattafiori, et al. The Llama 3 Herd of Models. arXiv preprint, 2024. doi
10. 45550//a2W..220..27775
[8] Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna B. Allal, Leandro Von Werra, and Martin Jaggi. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. In Advances in Neural Information Processing Systems, volume 38, pages 76232-76266 2024.
[9] Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Under- standing and Minimising Outlier Features in Transformer Training. In Advances in Neural Information Processing Systems, volume 38, pages 83786-83846, 2024.
[10] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-Key Normalization for Transformers. In Findings of the Association for Computational Linguistics: EMNLP, volume 2020, pages 4246 4253, 2020. doi: 10. 20111865311/2222 h0t.0.....niinggeemnlll....pmp
[11] Alejandro Hemándeez-anno al. Apertus: Democratizing Open and Compliant LLMs for Global Language Environments. arXiv preprint, 2025. doi: 10.48550/arXiw.25009 14233
[12] Jordan Hoffmann, et al. Training Compute-Optinal Large Language Models. In Advances Neural Information Processing Systems, volume 36, pages 30016-30030, 2022. ISBN 978-1-7138-7108-5
[13] Allen Hao Huang and Imanol Schlag. Deriving Activation Functions Using Integration. arXiv preprint, 2025. doi: 10..550aaii.22111333300
[14] Dhiraj Kalamkar, al. A Study of BFLOAT16 for Deep Learning Training. arXiv preprint, 2019. doi: 10.485500arxiv.I 19t.i...99........ 12322
[15] Jared Kaplan, et al. Scaling Laws for Neural Language Models. arXiv preprint, 2020. doi:
10. 10.48555/arx..20000..333
[16] Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks. In International Conference on Machine Learning, volume 41, pages 25333-25369, 2024.

---

[17] Dmitry Lepikhin, et al. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Representations, volume 2021.
doi: 10.1 1..1144583788036115555
[18] Ze Liu, et al. Swin Transformer V2:
Scaling Up Capacity and Resolution. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, volume 2022, pages 12009-12019) 2022.
[19] Iyy Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, volume 2019, 2019.
[20] Paulius Micikevicius, et al. FP8 Formats for Deep Learning, arXiv preprint, 2022. 10.48500ax1.22200055.4
doi:
[21] Team OLMo, et al. OL.o Furious. arXiv preprint, 2025. doi: 10.48550/arX1..5501 00066
[22] Karl Pearson. "Das Fehlergesetz Und Seine Verallgemeiine--nneen Durch Fechner Und Pearson." Rejoinder. Biometrika, 4(1-2):169-212: 1905. ISSN 0006-3444. doi
10. 10.109%/boomer/411211699
[23] Guilherme Penedo, et al. The WineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems, volume 38, pages 3081 1-30849, 2024.
[24] Houwen Peng, et al. FP8-LM: Training FP8 Large Language Models. arXiv preprint, 2023, doi: 10.4
[25] Qwen, et al. Qwen2.5 Technical Report. arXiv preprint, 2025. doi: 10.48550//XK1v2211111155
[26] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Opti- mizations Toward Training Trillion Parameter Models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, volume 2020, pages 1-16,
2020. doi:
[27] Bita Darvish Rouhani, et al. Microscaling Data Formats for Deep Learning. arXiv preprint, 2023. doi: 10.48350/axXv22310.000
[28] Noam Shazeer. GLU Variants Improve Transformer.
arXiv preprint, 2020.
doi:
10.48550/rxxw..00222co 05202
[29] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi Billion Parameter Language Models Using Model Parallelism. arXiv preprint, 2020.
doi: 10.48550/arXiv. 009.08853]
[30] Mingjic Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive Activations in Large Language Models. In First Conference on Language Modeling, 2024.
[31] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going Deeper with Image Transformers. In IEEE/VV international Conference on Computer Vision, volume 2021, pages 32-42, 2021. doi: 10.1 N0.1009CCCV48922222..00
[32] Ashish Vaswani, et al. Attention All You Need. Advances Neural Information Processing Systems, volume 31, 2017.
[33] Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-Loss-Free Load Balancing Strategy for Mixtureoof-Experts arXiv preprint. doi: 10-48550//rXx.2220855566
[34] Shuangfei Zhai, et al. Stabilizing Transformer Training by Preventing Attention Entropy Collapse. In International Conference on Machine Learning, volume 40, pages 40770-40803, 2023.
[35] Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without Normalization. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, volume 2025, pages 14901-14911, 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide in-depth empirical evaluations that support the claims made in the abstract in Section Section 4 discusses the utility of kurtosis to monitor FP8 training.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all of the details from our experimental setup needed to reproduce the main results. We also provide the implementation under anonymized repository.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: provide the code needed to reproduce the main experimental results. The dataset corpus used is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides/CCe httas//oooeeaaabsiiosoooooo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide the most important experimental details, including choice of
hyperparameters and infrastructure, in the experimental section. Further details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Training LLMs requires extensive computational resources, making it unfeasi- ble to perform repeated experiments under equivalent settings to obtain accurate statistical significance results. Nonetheless, our observations generalize across different scales and architectures.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detail our compute infrastructure in the experimental section. We further detail the different computational resources needed for each of the main experiments the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpa//////errp.cccpppppbee itw.iiieiiiiiiiieeeeeeeennee
Answer: [Yes]
Justification:
We have read and our experiments conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper addresses a topic on fundational machine learning research. While we provide a methodology accelerate language modeling learning and this brings many potential societal impacts, we feel they should not be specifically mentioned in the context of this project.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide proper citation of the code and data used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used in any meaningful way during the development of this project.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:s///ooripsccc/o/ooorrrrrree for what should or should not be described.
19

---

A Hyperparameters
We detail the selection of hyperparameters used in Table 6 For the case of FOG-flash, the og initialization value of tanh, entropy- regulariiggll is 0.5 for all model sizes. All models use linear warmup schedule, and I-sqrt cooldown schedule. The long-data 1.5B FOG-max experiment was trained for total of 430,000 steps, consuming approximately 450.9B tokens, using the same hyperparameters as the shorter run, including warmup steps.
Hyperparameter
390M
1.5B
8B
Layers (L)
16
16
32
Hidden size (D)
1024
2048
4096
FFN hidden size
4096
8192
14336
Attention heads
16
32
QK groups
4
8
Softmax scale* (s)
0.17678
0.125
Tied embeddings
Yes
No
Weight decay (A)
0.1
AdamW B1
0.9
AdamW Bg
0.95
Gradient clip value
1.0
Context length T
4096
Global batch size
128
256
512
Total training steps
100,000
125,000
10,000
Peak learning rate (1)
10-3
2.5 10
1.5 x 10
Warmup 1) steps
5,000
2,500
1,250
Cooldown steps
20,000
25,000
N/A
Minimum !
10
Table 6: Hyperparameters used in experiments. Note that FFN hidden size indicates the dimen- sionality of each linear projection gated activation functions; networks without GLUs 1.5x this value to match the parameter count. Softmax scale specified only applies to FOG models, all other models follow the standard 1//1///o /DQk.
B Architectures
We provide detailed formulations for all architectures presented this paper. Our transformer architecture consists of the following components in sequence:
1. Input token embeddings
2. An input scaling factor u E 00), which may equal 1
3. A series of T. transformer blocks as described below
4. A final normalization function (final), which may be the identity
5. A linear output layer
The transformer block is defined
block (X) = X+ (Nrfnoat)
FFN Nnfore) (X),
x ( GQA Ny(Dre)) (x).
The N²(*) are normalization layers that may be the identity, and FFN(X) is a two-layer FFN with nonlinear activation function and no bias. The GQA follows the standard grouped-query self- attention definition with softmax scaling factor s and Rotary Position Embeddings. Each attention head uses the definition
altnhead(X) == selfattn
(NO (NO) (XW(Q)), (o(K)) (XW(K)). (W(()))x((1)))
20

---

x > 0, if x < 0. where ((QK) is the reguliiire mechanism, and selfattn = PV. The P matrix is the attention probabilities matrix
= Softmax
(QQKT +M)
(1)
With this notation, Table 7 details the architecture families used in the project.
Parameter
((nnal)
(pre)
(post)
A(QK)
Llama3
Ng
N₂
id id
id
SwiGLU
Jlama3+SmoothhSviGL.
Nv
Nr
id
SmoothSwiGLU
OLMo2
Ng
id NG
Ng
SwiGLU
Op(a)
0
id
id
LayerScale,
N
GeLU
(OG-max
0 id
id Nr
W
xIELU
FOG-opr(ab)
0 id
id
N NG tanh
WV
GeLU
(OG-flash(ab)
0 id
id
GeLU
Table 7: Architecture details for the used models. Models with (a) initialize the post-normaiization gains with Yo = I/VT.. Models with (b) have frozen gains in the QK entropy regularization ((Q))) The id is the identity function, do is the chosen initialization standard deviation, . is the RMS normalization. The u input scaling is not trainable.
xIELU activation function Introduced in [3], the xIELU activation function defined element- wise as:
xIELU(+) == [ap=" 0.5x
(2)
(aa("" - 1) Og + 0.5x
where Op and on are trainable scalars per layer. xIELU is an extension of Squared ReLU and has been adopted and validated scale [11]
C Outliers impact quantization
Let's start with useful definition.
7-outlier: Given x E IRd and a = rms(x), the element x of x a 7-outlier if [x| 2 TO.
As increases, becomes larger outlier (a represents the natural magnitude of x). In practice, Before FP8 quantization, each tensor scaled with s(x) == MaxTPPSaauee/bbsmaxxxx to better utilize limited FP8 dynamic range.
т >>
Theorem 1. Let x E Id have T-outlier z, and x' E Rd have a r' oouuuirr z' with > т. Then for any subset T C (1...d) j, vector x will be quantized less accurately than Xr-
In other words, larger outlier values lead to less precise FP8-quantized results.
C.1 Proof
Let PPPMMxVVllue r//assaa(xx) and s' s(x').
Since absmax (x) 2 |ayl 2 ra, then S to (similarly, 8 S (.). Let m absmax(xx) and
m' absmax (x7).
Elements of Xy lie in [-m, m]. After scaling, the range becomes [-rm/(ra), rm/ in sxy, and [-rm/(r'o
(r' ()) rm//'()))) in s'x/.. Since r' T, (-rm/(r'o), ( C (rmm/(ro), rm/((ro) so x has smaller range.
This narrowed range contains fewer n-bit representable numbers, proving the theorem.
The proof applies to any subset T, including the set of "typical" values (e.g., 90%-quantile).
Thcorem guarantees large outliers worsen quantization on 90% of tensor elements.
21

---

C.2 Empirical confirmation
We measured activation values Llama and FOG-max 1.5B during mid-training on micro batch of data (precisely the second FFN layer's input, before quantization).
Observation: Llama presents 688-outlier while FOG-max shows only a 183-outlier. Using 90%- quantile, we get that 90% % of Llama's activation coefficients scale to [-0.289, 0.289] range, while FOG-max allows much broader range of [-2.084, 2.984].
D FOG extensions
In addition to our main experiments, we trained a 1.5B FOG model using the SwiGLU activation function, which label FOG-SwiGLU. This architecture was adapted from FOG-max, changing the activation function to SwiGLU and adjusting FFN hidden size to match the parameter count. Figure shows the loss progression of this model under FPSDPA training. This experiment resulted in completely stable training, and further demonstrates the flexibility of activation functions suitable our design.
FOG-SwiGLU
FOG
POG-SwiGl.
27
3.3
60
100
Cousuned Tolens
XIP
Figure 8: FOG-SwiGLU 1.5B FP8DPA run. FOG-max included reference. We observe stable training dynamics both approaches. The reported loss the 200-rolling-window mean and 5%-95% quantile bands.
MoE extension We adapted FOG-flash architecture to follow an MoE design, keeping the backbone configuration of the 390M model (hidden size, number of layers, etc), but upscaling with 8 FFN experts (2 active) following 117.. resulting in a 1.8B model, trained from scratch under the same configuration all 390M models. Additionally, and no explicit loss- balancannn loss, but rather an expert bias (3 with update rate of 0.01. We trained this model under BF16 and FPSDPA training, resulting
employ the z-loss [17] with coefficient of 0.01,
final converged loss of 2.477 and 2.483, respectively, as shown in Figure 9 This stable training result suggests good generalization capabilities for FOG other MoE designs under FP8DPA training. We further adapted our 1.5B models and similarly scale up to experts (2 active) to measure throughput gains at a larger scale. Table & summarizess our results, where FOG still provides the most throughput gains.
Throughput
Model
Precision
(tokens/sccond/GPU)
Llama3
BF16
3336
FP8
4202 (+24.8%)
FOG-flash
FP8DPA
4351 (+29.2%)
Table MoE training throughput. Measurements of 41B-SE MoEs taken using 4xGH200 nodes with expert parallel and pipeline parallel size of 4 using a batch size of 512. As with dense models, FOG-flash outperform all other architectures.
22

---

EPSDEA
0.4
2.4
Conennad Teienns
Figure 9: FOG-flas-MMoE 1.8B-8E FP8DPA loss progress. Both BF16 and FPSDPA trainings are shown. The FP8DPA training remains stable for the entire duration of training, and the final loss converged to both precisions remains within +0.005, suggesting comparable downstream capabilities. Loss reported is the (00-rolling-winde) mean and 5%-95% quantile bands.
E
FP8 training
In all our experiments, we used Transformer Engine's ' delayed scaling implementation with history length & = 1024 and margin O. Mathematically, given history of abs-max values, denoted H
[h+)== S [0, oo), of a tensor X, we define its scaling factor as:
FPSMax Value 2 max Hl
where FPSMaxValue E (0, 20) is the maximum value representable with the FP8 format used. We update the history using H < [max,ex |2/}] U [he)am to use for this activation in the next iteration. The end-to-end FP8 matrix multiplication is
GEMM(X, Y)
FPSGEMM(FPScaNT (p(X) (pSccat((p())) (Y))),
P (Y)
where FPSGEMM receives FP8 tensors and returns the BF16 result. We further detail the precision
used for every matrix multiplication during our FP8 and FP8DPA experiments in Table 9
Linear operators
Attention scores
Attention-vall value GEMM PV
Output layer
Method
QK
FP8
FP8
BF16
BF16
BF16
FP8DPA
FP8
FP8
FP8
BF16
Table 9: Comparison between FP8 methods. The FP8DPA method allows for all GEMM computations--excuu the output head- to be done with FP8 precision. In contrast, FPS training uses higher precision for the core attention computation. The linear operators are linear layers of the form Linearw (X) XW: namely the FFN linear layers, QKV projections and attention output projection. See Equation
I for the definition of the attention probability matrix P.
FP8 optimizer moments To further reduce memory usage, we tested FOG-flash under the usual FPSDPA setting, with an additional constraint: FP8 optimizer moments. This extends the typical setting of half-precision gradients and moments used most of our experiments. Figure TO shows the training loss across three different settings for comparison.
Fine-grained scaling recipes Recent FP8 training achievements, such as DeepSeek's DeemGEMM kernels, involve the use of fine-grained FP8 scaling recipes to provide a more robust training regime. While these options could potentially enable FP8 training when tensor-wise scaling alternatives diverge, it comes with significant overhead. We validate this claim using TransformerEngine's
23

---

10716
IURIPA
FUSDPA-I TosDrA-TP-0je
14-
100
1.5
Conemned
Figure 10: FOG-flash 390M loss curve comparing training precision. Our design reaches similar loss when trained with either precision. Loss reported is the 200-rolling-windov mean.
Blockwise scaling at the 8B scale. Training throughput reported in Table 10 Using FP8DPA training with delayed scaling recipe provides the highest boost across all tested methods.
Throughput (tokens/second/GPU)
Model
Precision
FP8 Recipe
Llama3
BF16
N/A
9.48k
Llama3
FP8
Blockwise
11.18k (+17.9%)
OP
FP8
Delayed
12.14k (+28.1%)
Llama--SmotthSviGLL
FP8
Delayed
13.1k (+38.2%)
FOG-flash
FPSDPA
Delayed
13.52k (+42.6%)
Table 10: Training throughput. Measurements taken using eight 4xGH200 nodes with Zero-l sharding (26), without model parallelism using a batch size of 1024. Notably FOG-flash outperform all other architectures.
F Long context
Note that when global batch size (GBS) increases -micro batch size fixed-, computation time takes over communication time. Therefore, the larger the GBS, the higher the throughput gains for all approaches: FOG-flash reaches +42.6% with GBS=lk compared to +40.2% with GBS=512. Moreover, enabling FP8 computations in the attention bring unique throughput benefits under long- context training. Hence the large efficiency gap achieved by FOG-flash FP8DPA compared to
Llama3+SmoottSwiiLLJ in Table 11
Context Length
TP
Llama3 (BF16) FOG-flash (FPSDPA)
Llama3-SmoottSwiGLU (FP8)
4096
1
9.48K
13.52K
13.13K
8192
1
9.08K
13.03K
Out Of Memory
8192
2
7.49K
10.42K
10.05K
16384
2
6.81K
9,45K
8.93K
Table 11: Training throughput under varying sequence lengths, performed at 8B scale using eight 4xGH200 nodes with global batch size of 1024. TP refers to Tensor Parallelism.
G
Evaluations
We selected the following set of benchmarks: ARC-Easy, Commonsense@AA HellaSwag, LAMBADA-OpenAA LAMBADA-standars OpenBookQA, PIQA. SocialIQA, and WinoGrande. We

---

used a standard open- LLM evaluation package for conducting these evaluations, cited in the code repository https:/ com/ alonyyouus4355534F0G
In Table 5 we report raw accuracy scores as percentages on three key benchmarks as well as the provide all scores along with their estimation errors for the 1.5B model size, demonstrating that the slight differences observed across many values statistically insignificant.
average the full set of tasks mentioned above. In Table 12
Architecture
Llama3
FOG-max
FOG-opt
FOG-flash
Hellaswag
43.7
43.3
43.4
43.3
12.7
42.8
11.9
+0.5
ARC-easy
71.8
71.6
73.0
71.3
70.8
70.9
69.4
10.9
PIQA
72.5
72.6
73.3
72.5
72.0
72.2
72.0
11.
Commonsense-qa
19.6
20.2
22.2
19.3
21.2
21.1
20.8
#1.2
Lambada-openai
44.5
43.7
44.6
44.3
44.5
42.4
41.2
#0.7
Lambada-stane standard
38.9
37.0
39.5
37.9
35.9
33.8
=0.7
Openbook-on
26.2
28.0
27.0
26.8
27.0
28.4
27.6
+2.0
Social-iqa
41.3
11.8
42.0
11.7
40.8
41.0
40.8
+1.1
Winogrande
56.5
55.6
58.7
54.6
57.4
56.9
56.8
11.4
Average
16.1
46.0
47.1
45.7
46.0
45.7
44.9
#0.3
Table 12: More detailed results 1.5B scale. For each model and each task, the first score results from BF16 training and the second from FP8DPA training.
Weight decay cooldown As mentioned Section we experimented with cooling down the weight decay, often used constant value equal 0.1 that is coupled with the learning rate, see if it solves the OP+frozenQK architecture's consistent divergence during the learning rate decay phase. We also tested it other architectures and, optimize the of resources, for the final experiments. This trick helped stabilize the weights' norm indeed, but couldn't It solve the divergence issue. Moreover, had effect on final performance on stability. Table 13 highlights this no-effect claim at 1.5B scale.
had to keep it later
Setting
WD
Loss
Average score
OP+FrozenQK cooldown diverges
OP+Frozen
constant diverges
FOG-opt
cooldown converges
46.0 +0.3
FOG-opt
constant
converges
46.3 +0.3
Table 13: Weight Decay (WD) during the LR decay phase. constant, equals 0.1. Else, it starts from 0.1 and is proportional to LR.
25

---

H
Computational Resources
Our experiments were conducted on nodes equipped with Grace Hopper (GH200) GPUs each. We typically used 4, 8, and 16 nodes for our 390M, 1.5B, and 8B parameter experiments, respectively, with minor variations across different runs. Importantly, all throughput measurements were taken under identical hardware configurations. Table [4]diiiiss the computational resources in GPU hours (GPUh) required for our main experimental results. This includes the computational cost of training all architectures that diverged during FP8DPA training, the FP8DPA and BF16 stable training runs for our three main architectures, and the BF16 Llama3 baseline. The aggregation includes node start-up times, computation lost due to node failures, and overhead from calculating and logging kurtosis metrics. The complete research project required additional computational resources beyond those conducted numerous preliminary experiments and explored ideas that did
specified in the table, not appear in the final paper.
Group
GPUh
Divergent runs (FP8DPA)
886
Llama3 baselines (BF16)
1,395
FOG experiments
11,162
Table 14: GPU hours used for the main experiments.
26


Reviews:
[{"review_id": "REcuYKR9q7", "reviewer": "Reviewer_CRBu", "summary": "This paper introduces FOG (Fast and Outlier-Guarded), a novel set of LLM architectures specifically designed to enable stable FP8 training with all GEMMs, including attention computations (FP8DPA). The authors address a critical limitation in current FP8 training approaches that either use suboptimal fine-grained scaling or fall back to higher precision in sensitive components. The key innovation is architectural modifications that reduce activation outliers, enabling the use of simple delayed scaling strategies while achieving throughput improvements of up to 40%.", "questions": "See weaknesses", "limitations": "See weaknesses", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n### 1. **Significant Technical Achievement**\nThis is the first work to successfully demonstrate stable FP8 computation in attention mechanisms at scale. The ability to use FP8 for all GEMMs within transformer blocks represents a meaningful step toward fully low-precision LLM training.\n\n### 2. **Comprehensive Empirical Validation**\n- Extensive experiments across multiple model scales (390M, 1.5B, 8B parameters)\n- Long-term training validation (up to 420B tokens, 14× Chinchilla optimal)\n- Thorough comparison with existing architectures shows consistent divergence in competitors\n- Downstream performance matching or exceeding BF16 baselines\n\n### 3. **Practical Impact**\n- Substantial throughput gains (35-40%) with maintained model quality\n- Uses simple delayed scaling rather than complex fine-grained approaches\n- Provides actionable architectural guidelines for the community\n\n### 4. **Novel Diagnostic Tool**\nThe use of kurtosis as an early warning system for FP8 training divergence is innovative and potentially valuable for future research. The observation that kurtosis exhibits abnormal behavior well before loss divergence could be broadly applicable.\n\n### 5. **Thorough Ablation Studies**\nThe step-by-step transition from OP to FOG architectures clearly identifies the critical components (frozen QK gains, post-normalization) responsible for stability.\n\n## Weaknesses\n\n### 1. **Definition of xIELU**\nWhile xIELU is a new activation function that is not commonly used, I believe a definition is necessary in this paper for readers to understand it.\n\n### 2.  **Further Explanation of the FP16 baseline**\nThe proposed method entails several architectural changes compared to the existing LLM architecture in LLaMA3. Therefore, I would expect the authors to provide more comparisons between the two architectures in the experiments. Moreover, I still do not understand why it can match the original FP16 LLaMA3 with pre-norm and SwiGLU.\n\n### 3.  **Loss Drop of Llama3 in Figure 7**\nFrom Figure 7, the llama3 baseline has a much sharper loss drop after 100B tokens. Can the authors provide further explanation for this phenomenon?  Moreover, why is this drop not a good feature, but some other feature with worse smoothness?\n\n### 4. **Lack of ablation**\nI would expect some ablation for llama3 to the new architecture in FP16, demonstrating how the architecture itself affects performance.", "was_revised": true, "final_justification": "During the discussions with the authors, the problem mentioned in the initial reviews has been mainly resolved, with confirmation of updating in the next version. I believe this work offers a comprehensive framework for FP8DPA training. Based on this information, I have raised my score.", "rebuttal": "We thank the reviewer for their time and thoughtful feedback.\nWe are glad they found many aspects of the work promising.\nBelow, we address the questions and concerns raised.\n\n> Definition of xIELU\n\nAlthough we provide the reference work [1] that introduces xIELU in line 186 of the submitted manuscript, we agree that this non-linearity has been proposed relatively recently. And adding its definition to the manuscript would give readers a clearer picture of FOG-MAX variant. We thank the reviewer for pointing this out and will provide a clear definition in the appendix upon acceptance.\nAs we mention xIELU, let's note that this choice was deliberate. It's an inherently quadratic function and thus is an outlier-amplifier. Therefore, choosing an architecture with xIELU while demonstrating that our method still remains very stable highlights FOG's significant robustness to outliers, pointing to the limits of the excellent former work [2], as discussed in paragraph: L183-L191. \nWe believe that the FOG's robustness when using quadratic activation functions is an important strength of our work that we should make clearer in the next revision.\n\n> Loss Drop of Llama3 in Figure 7\n\nThe loss drop pointed out in Figure 7 is expected, and matches the LR cooldown during the last phase of the widely adopted WSD LR schedule.\nThis phenomenon is positive and allows matching cosine LR schedule's final performance [3]. Note that the drop is consistently observed for all architectures reaching the final LR cooldown phase, such as around 40BT in Figure 2; around 40BT for the 390M model and 100BT for the 1.5B model in Figure 6.\nFor the long FOG-Max-1.5B run trained on over 420BTs, LR cooldown hasn't been applied yet as the only goal was showcasing stability. To address the reviewer's concern, we've pursued this run by **conducting an extra 25BTs of LR cooldown**, which resulted in an expected loss drop reaching a final value of **2.27**, improving upon Llama-1.5B final loss in Figure 7.\n\n> Further Explanation of the FP16 baseline + Lack of ablation\n\nWe first clarify that BF16 is the standard half-precision format used for training [4]. Although model weights can be stored in FP16, GEMMs are done in BF16 for more dynamic range. In the following we assume the reviewer refers to this widely adopted setting, that we call BF16 training. \n\n*\"I would expect the authors to provide more comparisons between the two architectures in the experiments\"*:\nWe kindly refer the reviewer to Table 4 of the supplemental materials, where we report a wide variety of **downstream performance** results following extensive experiments to compare FOG variants against the Llama BF16 baseline. We note that for each task and each FOG variant, downstream performances are provided for both BF16 training and FP8DPA training. We also point to Figure 4 where **upstream performance (loss)** aligns with task performances, and to Table 4 where we report **throughput comparisons**.\n\n*\"I still do not understand why it can match the original FP16 LLaMA3 with pre-norm and SwiGLU\"*: \nTable 4 shows that all BF16 FOG-variants exhibit similar performances to original BF16 Llama3. Indeed, FOG is designed to preserve both stability and performance when reducing GEMMs precision to FP8, even when FP8 computation of the dot product attention (FP8DPA) is carried out. This is highlighted by the scores reached with FP8DPA training as well as the equivalent final losses. We explain this \"equivalent performance with both BF16 and FP8DPA\" by **the robustness of the architecture** as well as the choices of **expressive activation functions** and beneficial **QK-regularization**.\nFurther, we note some consistent trends such as FOG-max slightly outperforming other architectures, including baseline. In [1], it has been shown that xIELU outperforms SwiGLU thanks to its linearly increasing gradient for positive inputs and its trainable gradient for negative inputs. QK-RMSNorm and QK-Tanh play a role of entropy regularizers, preventing spiky attention softmax outputs, crucially contributing to FOG's performance and stability. In the following table where we introduce two additional experiments, we re-confirm the stabilizing effect of QK regularization under FP8DPA regime, by the delayed divergence. The starting point of the architecture ablated below is now Llama and QK is introduced after Post-LN.\n\n| Size=390M (FP8DPA)     | Data Budget | Normalization | QK-RMSNorm | Activation Function   | Observation                    |\n|-------------------------|-------------|----------------|---------|----------------------|----------------------------------|\n| Llama3           | 50B tokens  | Pre-LN         | No      | SwiGLU                      | Early divergence (Fig-6)         |\n| Llama3 w/ Post-LN        | 50B tokens  | Post-LN        | No      | SwiGLU              | Early divergence (new)           |\n| oLMo                    | 50B tokens  | Post-LN        | Yes     | SwiGLU               | Mid-training divergence (Fig-6)  |\n| FOG-SwiGLU-1.5B**              | 125B tokens | Post-LN        | Yes *   | SwiGLU        | Convergence, equivalent to baseline (new) |\n\n*Frozen QK-norm gains.  \n**Note that the FOG-SwiGLU variant test scales up model size and data budget compared to the other 390M architectures.\n\nTo conclude, our work shows that FP8 precision is sufficient to reach the long standing half-precision performances **without trade-offs**, while providing **flexible architectural choices** (QK-regularizers, sub-linear/quadratic/gated activations) with a training **acceleration that scales better with long context** than all previous approaches, thanks to the FP8 attention computation.\n\n\n[1]: Allen Hao Huang, Imanol Schlag, 2025, Deriving Activation Functions Using Integration.  \n[2]: Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry, 2025, Scaling FP8 training to trillion-token LLMs, in The Thirteenth International Conference on Learning Representations.  \n[3]: Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Benallal, Leandro Von Werra, Martin Jaggi, 2024, Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations, in The Thirty-eighth Annual Conference on Neural Information Processing Systems.  \n[4]: Dhiraj Kalamkar, et al., 2019, A Study of BFLOAT16 for Deep Learning Training."}, {"review_id": "IzcPGBwEWp", "reviewer": "Reviewer_r2ET", "summary": "This paper proposes an FP8 LLM training method.", "questions": "See weakness.", "limitations": "See Weakness.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This paper proposes a pure FP8 LLM training method\n2. Table 4 shows that FP8 significantly improves throughput\n\n**Weaknesses**\n\n1. The current evaluation scenario is mainly short text scenarios. It is unclear whether FP8 will cause serious value problems in long text scenarios.\n2. The volume of data validated in the paper is insufficient. While the authors argue that this aligns with the optimal cost-performance tradeoff, practical implementations (e.g., Llama and DeepSeek) typically employ pre-training datasets that far exceed this \"optimal\" threshold. Moreover, as stated in Section 4.3 of [1], a 1B-parameter model with FP8 requires training on approximately ~27T tokens to reliably observe significant degradation caused by quantization.\n3. The paper appears structurally incomplete, and crucially, key appendixes are missing.\n\n**Reference**\n\n[1] Sun, Xingwu, et al. \"Scaling Laws for Floating Point Quantization Training.\"", "was_revised": true, "final_justification": "The authors have provided experimental evidence that sufficiently clarifies the loss behavior in extended-context scenarios. While I maintain that the scale of validation and longer context results require more extensive verification, I will recommend this paper for publication.", "rebuttal": "We thank the reviewer for their time.\nHowever, we believe several of the concerns raised do not accurately reflect the contributions of our work.\nWe address these points below with clarifications and supporting evidence.\n\n> Short text scenarios\n\nWe thank the reviewer for pointing out to larger context length scenarios.\nFirst, let's note that the sequence length of 4096 adopted in our experiments lies within the standard configurations of current small and large scale language model pre-training setups, such as DeepSeek-v3 [1], OLMo2 7B,13B,32B [2], Qwen2.5 0.5B-72B [3], and SmolLM3 3B [4].\nThis moderate context-length pre-training stage usually takes the largest amount of time, mainly because of the prohibitive (quadratic) cost of attention. For instance, Qwen3 models were trained with sequence length 4k for more than 35T tokens before switching to longer values for a few hundred billion tokens. It took only 100B tokens out of 11.3T tokens for SmolLM3 to perform the final long context expansion phase. Finally Llama3.1-405B trained on long sequences for only 0.8T tokens out of nearly 16T tokens in total.\nHence, speeding up the main moderate-context phase of pretraining has the biggest impact on overall training efficiency.\n\nNonetheless, long context training after the main phase can still be relatively expensive because iterations become considerably slower.\nOur method benefits from longer contexts, because we perform attention computations in FP8.\nIndeed, attention tends to dominate the computational cost of the forward-backward pass **as sequence length increases**, which intuitively **yields even larger gaps compared to previous FP8 approaches**.\n\n**We demonstrate this trend at 8B model scale** by benchmarking and report throughput values (in tokens/second/gpu) results in the following table:\n\n| Sequence Length | LLaMA BF16 | FOG FP8DPA | Smooth-SwiGLU FP8 |\n| --- | --- | --- | --- |\n| 4K  (TP=1) | 9.48K | 13.52K (**+42.6%**) | 13.13K (**+38.5%**) |\n| 8K  (TP=1) | 9.08K | 13.03K (**+43.5%**) | Out Of Memory |\n| 8K  (TP=2) | 7.49k | 10.42K (**+39.1%**) | 10.05K (**+34.2%**) |\n| 16K (TP=2) | 6.81k | 9.45K (**+38.8%**) | 8.93K (**+31.1%**) |\n\nA batch size of 1024 was used for all experiments, providing even better efficiency gains than the ones reported in the paper (+40.2% --> +42.6%).\nActually, when GBS increases, computation time takes over communication.\nClearly, the gap between the FP8-only approach (Smooth-SwiGLU) and our FP8DPA setting becomes larger as context increases.\nTo be more specific, we do not claim higher accelerations with higher context lengths because such settings generally require more model sharding, resulting in lower boosts. But we do show intuitively and empirically that the gap with existing FP8 methods increases with the context length: which is a strength that we'll make sure to highlight in the next revision.\n\nFinally, we performed an **additional experiment** and ablated Llama BF16 and FOG-max FP8DPA 390M-sized models, for 50B tokens **with a much longer sequence length of 16k**, using the same configuration as specified in our paper at this scale (Table 1 of the Appendix).\nThe results were consistent with those reported at 4K sequence length: stable loss progression and equivalent final performance. Therefore there are no *\"serious value problems\"* with longer sequence length training, but instead the opposite of even stronger efficiency gains.\n\n> The volume of data validated in the paper is insufficient\n\nWe would like to present the following 3 arguments to answer this point:\n- **Practical implementations** of state-of-the-art models such as DeepSeek-v3-671B scale data to a point that corresponds to roughly 1.2 x the compute-optimal data budget taken from the chinchilla rule of thumb [5]. The factor is around 11x for Llama-3.1-70B and even less for Llama-3.1-405B. Therefore, we still do believe that scaling training of a 1.5B model to 420 billion tokens, which is 14x the compute-optimal, can be considered as a long data regime at this scale. Especially, given the consistency in performance and stability we've demonstrated when scaling model size.\n- One of this paper's novelties, is the use of kurtosis to monitor and potentially predict future divergences. In Figure 4, we have provided **low kurtosis guarantees** for much longer than the above mentioned practical chinchilla factors, as well as a **sublinear progression** further strengthening the stable data scaling argument.\n- Judging from most of the practical implementations, scaling the training of a 1B model to 27T tokens is almost never done. Further, distillation seems to be the go-to recipe in general to obtain cheap but strong small language models (cf. Gemma3, Falcon3). \n\n> The paper appears structurally incomplete, and crucially, key appendixes are missing\n\nThank you for your comment.\nWe would have appreciated more specificity and clarity regarding the \"structural incompleteness\". There is a misunderstanding, as our appendix was already provided in the standard supplementary material submission as instructed by NeurIPS, and was always available.\nRegarding the reviewer claims about \"missing appendices\":\n- \"Section 5.2: Claims weight decay analysis exists 'later in the Appendix'\".\n    - We kindly redirect the reviewer to **lines 41-46 and validation experiments represented by Table 5** of the supplemental materials uploaded in addition of the main manuscript, where such analysis can be found.\n- \"Section 5: Promises hyperparameter details 'available in the Appendix'\".\n    - These details are described in **lines 1-6 and clearly represented in Table 1 of the Appendix**, present in the supplemental materials uploaded in addition of the main manuscript.\n\n> Table 4 Precision Labeling Inconsistency\n\nWe appreciate the reviewer for pointing out the \"BF6\" typo in Table 4. We'll make sure to correct it in the next revision.\nHowever, although we understand the reviewer might have been influenced by the misbelief of \"missing appendix\", we'd still like to raise our concerns about overemphasising over a clear typo. We'd like to point the reviewer to the comment made by reviewer 5MCk: \"This paper is well written, logically clear, and fully expresses the motivation and methods\".\n\nWe thank reviewer r2ET for the feedback.\nAs we addressed all of the concerns, we hope it is now clearer that the paper is technically sound and that no parts are missing. We kindly ask the reviewer to adjust the scores accordingly.\n\n[1]: DeepSeek-AI, et al., 2025, DeepSeek-V3 Technical Report.  \n[2]: Team OLMo, et al., 2025, 2 OLMo 2 Furious.  \n[3]: Qwen, et al., 2025, Qwen2.5 Technical Report.  \n[4]: Elie Bakouch, et al., 2025, SmolLM3: smol, multilingual, long-context reasoner.  \n[5]: Jordan Hoffmann, et al., 2022, Training Compute-Optimal Large Language Models, in Proceedings of the 36th International Conference on Neural Information Processing Systems."}, {"review_id": "Yp5qlZkMSF", "reviewer": "Reviewer_nVZB", "summary": "This work enables fully FP8 training of large language models (LLMs), including all GEMMs, even in attention layers (termed FP8DPA), to improve training throughput without sacrificing performance. It proposes the FOG architecture with changes in norm and activation layers. Besides, the kurtosis metric is proposed to track the outliers in the activations to diagnose/predict potential divergence behavior. This work leads to up to 40% speedup compared to BF16 baselines.", "questions": "There's a statement in L34 that multiple scaling factors per single tensor\"comes with an efficiency overhead\". Do you have benchmark result to demonstrate how much slower it is? I believe the deepseek deepgemm kernels are already heavily optimized with cutlass and low level primitives.", "limitations": "- FOG’s techniques may not generalize to all future transformer variants. Specifically, it does not test the MOE model family, and it's unclear if it's activation change would affect the performance of MoE models. \nlimitations mentioned by the author:\n- FP8 optimizer states not yet explored.\n- it still uses higher precision for the final projection layer due to sensitivity.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n- the first work to enable fp8 training on attn (and hence all GEMMs)\n- the kurtosis is a novel, insightful metric to predict outlier and potential divergence issues, with experimental result and analysis done on fog and non-fog model recipes.\n- addresses an important problem since fp8 dtype is available in mainstream accelerators from AMD and NVIDIA\n- reasonable speedup and convergence result\n\nWeakness:\n- the benefit of fp8 attention is not clarified. It does not show breakdown analysis on how much efficiency benefit fp8 attn brings, compared to other fp8 baselines such as OP. It would be good to justify the motivation to run fp8 attn.\n- In this work, only 4k context window is used. Due to computation pattern of attn, the longer the sequence length, the more flops it consumes (hence more benefit if it's done in fp8). However, this work does not explore sequence length longer than 4k, which brings questions on whether the technique is useful for scenarios where bf16 attention computation is truly the bottleneck of the training. Furthermore it is fundamentally harder to run fp8 attention with longer sequences due to numerical stability issues, which I'd like to see it addressed. \n- more rigorous benchmark is needed. The evaluation datasets (hellaswag/ARC/etc) are relatively simple. More comprehensive evaluation would make this work more convincing.\n- Lack of clarify in table 4. Does Llama+SmoothSwiGLU mean bf16 attn? If not, there is missing reasonable fp8 baseline that converges. For instance, assuming deepseek-v3 fp8, and llama+SmoothSwiGLU+bf16 attn are expected to convergence, and FOG should be comparing with these converging fp8 baselines, on the efficiency gain. If Llama+SmoothSwiGLU does use bf16 attn, then it brings another question that fp8 attn does not bring much actual benefit (efficiency) compared to already working fp8 recipes. \n- limited generality with regard to other architectures, such as MoE", "was_revised": true, "final_justification": "The additional experiments addressed most of my concerns:\nFP8 optimizer states - addressed\nHigher precision for the final projection layer - confirmed as limitation \nLimited generality with regard to other architectures - partially addressed with preliminary results \nSequence length beyond 4k - addressed \nLack of clarity in Table 4 - addressed", "rebuttal": "We thank the reviewer for the submitted review and weaknesses highlighted.\nPlease find next a detailed answer for each point raised next.\n\n> The benefit of fp8 attention\n\nThe main advantage of FP8DPA training in our work is the significant throughput boost compared to existing methods, as shown in Table 4.\nMoreover, we kindly refer the reviewer to the answers of the next points in the current rebuttal, where we include new experiments comparing other FP8 recipes, and efficiency on longer sequence lengths.\nWe believe these additions further strengthen the advantages of FP8DPA training.\n\n> Sequence length beyond 4k\n\nWe thank reviewer for this excellent point.\nFirst, we note that the sequence length of 4096 adopted in our experiments lies within the standard configurations of current small and large scale language model training setups such as DeepSeek-v3, OLMo2, Qwen2.5, and SmolLM3.\nThis pre-training stage usually takes the largest amount of time, mainly because of the attention's quadratic complexity. For instance, only ~100B out of 11.3T of SmolLM3, and 0.8T out of the nearly 16T of Llama3.1 405B tokens were consumed in the long-context extension training phase.\nHence, it matters the most to speed up this compute-consuming moderate-context phase of training.\n\nNonetheless, long context training phase can still be relatively expensive.\nTherefore, by safely performing attention computation in FP8, our novel approach benefits the most from context extension.\nIndeed, attention tends to dominate the computational cost of the forward-backward pass as sequence length increases, which leads to even larger efficiency gains when using FP8DPA compared to previous FP8 approaches.\n**We demonstrate this trend at 8B model scale**  in the following throughput (tokens/second/gpu) table:\n\n| Sequence Length | LLaMA BF16 | FOG FP8DPA | Smooth-SwiGLU FP8 |\n| --- | --- | --- | --- |\n| 4K  (TP=1) | 9.48K | 13.52K (**+42.6%**) | 13.13K (**+38.5%**) |\n| 8K  (TP=1) | 9.08K | 13.03K (**+43.5%**) | Out Of Memory |\n| 8K  (TP=2) | 7.49k | 10.42K (**+39.1%**) | 10.05K (**+34.2%**) |\n| 16K (TP=2) | 6.81k | 9.45K (**+38.8%**) | 8.93K (**+31.1%**) |\n\nA batch size of 1k was used for all experiments, providing even better boosts than the ones reported in the paper (+40.2% --> +42.6%).\nClearly, the gap between the FP8-only approach (Smooth-SwiGLU) and our FP8DPA setting becomes larger as context increases.\nWe note that training with even longer contexts usually relies on strategies like ring attention, which bring back the sequence length processed per GPU to values closer to 16K. We thus limit ourselves to ablations of up to 16K.\n\nTo address the concern about FOG's robustness in such long context scenarios, **we conducted two additional experiments** (FOG-Max FP8DPA; Llama BF16) at 390M scale for 50BTs, using **a sequence length of 16384**. \nResults can be summarized as follows: A similarly smooth loss progression with an equivalent final performance, around ~2.28 for FP8DPA FOG and 2.22 for BF16 Llama. Such a small loss gap may not translate to downstream performance, especially considering the possibility of running FOG long-context training for +38.8% more tokens at the same FLOP budget.\n\n> More rigorous benchmark is needed\n\nWe understand the reviewer's concern about the \"easy\" benchmarks used in the evaluation.\nWe first draw the attention of the reviewer to Table 4 of the appendix, where a wider set of benchmarks is detailed.\nWhile most of these are known to be \"early signal\" tasks, that is exactly the reason why we chose them:\n- The scale of 390M models trained on 50BTs and 1.5B models on 125BTs is **not large enough to expect for advanced capabilities to emerge**. Therefore, evaluating on \"harder\" tasks would result in close-to-random scores.\n- The selected tasks evaluate a **wide arrange of skills** (e.g. hellaswag for common-sense inference, OpenBookQA for assessing understanding of different subjects, etc).\n- A quick analysis of the scores shows that these **benchmarks are far from being saturated** by the trained models while exceeding the random performances. Thus, the comparison is fair for the scale.\n- Prior work [1] report performance on a similar suite of benchmarks, supporting their relevance in this context.\n\n> Lack of clarity in Table 4 | Granular FP8 recipes overhead\n\nThe throughput reported in Table 4 for Llama+SmoothSwiGLU indeed utilizes BF16 computation in the attention, as enabling FP8 attention results in an early divergence shown in Figure 6.\nAs explained in the Introduction and furthered detailed in Table 3 of the appendix, we associate the label \"FP8\" to the setting where attention is in BF16 and label \"FP8DPA\" the settings were FP8 attention computation is used.\nNext, **we include simple FP8 training with OP architecture and granular FP8 approaches** as these were not shown to diverge  and **remain viable FP8 candidates**.\nWe compared the throughput of a set of 8B models with the same batch size (1k) across 8 nodes in the following table.\n\n| Architecture | Precision | FP8 recipe | Throughput (tokens/sec/gpu) |\n| :- | :-: | :-: | :-: |\n| Llama | BF16 | N/A | 9.48k |\n| Llama | FP8 | Blockwise* | 11.18k (+17.9%) |\n| OP | FP8 | DelayedScaling | 12.14k (+28.1%) |\n| Llama+SmoothSwiGLU | FP8 | DelayedScaling | 13.1k (+38.2%)  |\n| FOG-flash | FP8DPA | DelayedScaling | 13.52k (+42.6%) |\n\n\\* NVIDIA's Transformer Engine implementation.\n\nOur approach still yields the highest efficiency.\nResponding to the concern that \"fp8 attn does not bring much actual benefit (efficiency) compared to already working fp8 recipes\", we argue, as motivated in the lines 33-35 of the paper and further evidenced by the table above, that using granular scaling factors such as  NVIDIA blockwise recipe, provides sub-optimal throughput gains. Such FP8 kernels are already heavily optimized but the overhead is inevitable from such a scaling strategy.\n\nMoreover, while the extra ~4.4% throughput boost compared to the fastest competing FP8 alternative (Llama+SmoothSwiGLU) could be seen as small, the proposed FOG architectures still have several advantages compared to it.\nOur FOG architecture family admits a **several QK regularization functions, and flexible choice of activation functions**, generalizing even to MoEs as demonstrated next.\nMoreover, the benefits of FP8DPA training become **more significant as the sequence length grows**, as shown previously, making our recipe more valuable during long-context training.\n\nIn addition of GeLU and xIELU activations validated in the paper, **we additionally experimented with a 1.5B FOG-max FP8DPA variant with SwiGLU activation**, which we train for 125B tokens.\nThis new experiment resulted in a **successful convergence**, similar to the BF16 baseline.\nMoreover, the choice of GeLU and xIELU activations in the proposed variants was deliberate and further validates that FOG can generalize to other activations. xiELU is a quadratic function and thus amplifies input outliers.\nTherefore, succesful FP8DPA training highlights FOG's robustness to outliers, and draws new limits of previous work [1], as discussed in lines 183-191.\nMoreover, the SmoothSwiGLU trick is not enough for stable FP8DPA training.\nWe believe that allowing the use of quadratic activation functions with FP8DPA training is an important novelty of our work that we should make clearer in the next revision.\n\n> Limited generality with regard to other architectures\n\nThe architecture suite proposed encompass various architectural choices with different normalization and activation function choices, as furthered evidenced in the previous point.\nWhile future transformer variants are impossible to predict fully, we provide state-of-the-art results for contemporary architectures and allow for flexibility within our architecture.\nMoreover, a key contribution of our work highlights the importance of kurtosis and outlier minimization for FP8 training monitoring, bringing a valuable tool to the scientific community to diagnose FP8 acceleration when such future transformer variants are proposed.\n\nWhile we don't provide extensive MoE results, we expect FOG to generalize well. Aside from the routing mechanism, MoEs follow similar forward passes as dense models. Our key changes, post-norm and QK regularization, don't interfere with MoE-specific components, and new FOG-SwiGLU stable variant further supports compatibility of our appraach.\nStill, **we adapted FOG-flash-390M to an MoE with 2 active experts among 8** (resulting in a 1.8B model). We trained it for 25BTs, following the experimental setup in our paper, with both FP8DPA and BF16 precision.\nThe resulting 1.8B-8E FOG model converged to a loss of **2.477** with BF16, and **2.483** with FP8DPA. Which **suggests good generalization of FOG for MoEs under FP8DPA regime**.\n\nFinally, **we performed throughput measurements on a 41.5B-8E model**.\nAs with dense models, our architecture provides the largest boost compared with other approaches.\n\n| Architecture | Precision | Throughput (tokens/sec/gpu) |\n| :- | :-: | :-: |\n| Llama MoE | BF16 | 3366 |\n| Llama+SmoothSwiGLU MoE | FP8 | 4202 (+24.8%)  |\n| FOG-flash | FP8DPA | 4351 (+29.2%) |\n\n> FP8 optimizer states\n\nIn addition to the points discussed in the Limitations section, **we ran a new FP8DPA experiment** using FOG-flash at 390M scale **with FP8 optimizer moments**, following the setup in our paper.\nThe model converged successfully to a loss of **2.645**, nearly identical to **2.649** obtained with higher-precision moments.\n\n> Higher precision for the final projection layer\n\nAs mentioned in the Limitations section, we decided to explore this further in a future work.\nWe believe the final projection layer is the most sensitive part of the architecture.\n\n[1]: Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry, 2025, Scaling FP8 training to trillion-token LLMs, in The Thirteenth International Conference on Learning Representations."}, {"review_id": "yxvZFenyX7", "reviewer": "Reviewer_5MCk", "summary": "This paper mainly addresses the stability issue of FP8 during the LLM pre training phase. The paper implemented FP8 training on Self attention and FFNs, and compared it with previous methods. Extensive experiments have demonstrated the effectiveness of the method. The core goal is to eliminate outlier issues that occur during the FP8 stage.", "questions": "See weaknesses.", "limitations": "N/A", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper is well written, logically clear, and fully expresses the motivation and methods of the paper.\n2. The paper conducts extensive experiments, including comparative experiments on models of different types and scales. \n3. The authors provide a wide range of comprehensive evaluation metrics.\n\n**Weaknesses**\n\n1. Firstly, the study lacks necessary ablation experiments, such as the different components listed in Table 1. The author only provided conclusions without relevant ablation experiments. At the same time, there is a lack of more possibilities, such as activations outside of GeLU.\n\n2. Secondly, for the modification of transformers, there are more enlightening and lack additional explanations for replacing or deleting different components.\n\n3. Finally, the core purpose of the article is to eliminate the negative impact of outliers. The author only presented the final results and lacked a key explanation for eliminating outliers. If there could be a more reasonable explanation, it would enhance the completeness of this article.", "was_revised": true, "final_justification": "I have re-examined this paper and will stand by my previous comments.", "rebuttal": "We thank the reviewer for their feedback and hope the concerns will be fully addressed with the following response.\n\n> The study lacks necessary ablation experiments | The modification of transformers\n\nWe would like to point the reviewer to the Section 3 (after Table 1) as well as Section 4, where we present **extensive ablations and a step-by-step transition** from OP architecture, that faces early divergence, to FOG-Max variant. We further **validate with kurtosis observations** (Figure 3) to isolate the components causing instability or outlier amplification. Including Section 5, we have **ablated around 8 architectures, covered 3 model sizes (0.4B, 1.5B, 8B) and different data regimes (50BTs, 125BTs, 420BTs)**. We would also like to quote the reviewer 5MCk: \"The paper conducts extensive experiments, including comparative experiments on models of different types and scales\" and \"The authors provide a wide range of comprehensive evaluation metrics\".\n\nTo further address the reviewer's concern, **we conducted additional experiments, this time starting from Llama architecture and reaching a new variant we call FOG-SwiGLU**. In this study, QK norm is introduced after Post-Layernorm, considerably delaying divergence. The delayed divergence confirms the benefits from its entropy regularization effect, avoiding peaky softmax outputs. Finally, Freezing QK ensures convergence while matching the Llama3 BF16 baseline.\nThe following table presents further details about the study:\n\n| Size=390M (FP8DPA)     | Data Budget | Normalization | QK-RMSNorm | Activation Function | Observation                      |\n|-------------------------|-------------|----------------|---------|----------------------|----------------------------------|\n| Llama3                  | 50B tokens  | Pre-LN         | No      | SwiGLU               |    Early divergence (Fig-6)        |\n| Llama3 w/ Post-LN        | 50B tokens  | Post-LN       | No      | SwiGLU               |   Early divergence (new) : lower kurtosis |\n| oLMo2                     | 50B tokens  | Post-LN       | Yes     | SwiGLU               | Mid-training divergence (Fig-6)  |\n| FOG-SwiGLU-1.5B**        | 125B tokens | Post-LN       | Yes *   | SwiGLU               | Convergence, equivalent to baseline (new) |\n\n*Frozen QK-norm gains  \n**Note that the FOG-SwiGLU ablation scales up model size and data budget compared to the other 390M architectures.\n\nReviewer also mentioned that “ there is a lack of more possibilities, such as activations outside of GeLU “.\nWe remind that FOG-Max, a key variant among the initially proposed FOG architectures, uses **xIELU activation function** which belongs to the family of quadratic and learnable activation functions. The above-mentioned additional study, further validates the stability of a 4th variant that simply keeps **SwiGLU gated activation function**. This demonstrates unprecendented generalization, unlike prior work which is tied to a scaled version of gated activations\n\n> Lacked a key explanation for eliminating outliers\n\nThe core motivation of eliminating outliers stems from the fact that, because of the limited dynamic range of FP8 formats, the presence of large outlier features increases the risk of under- or over-flowing values.\nThis is motivated in lines 26-32, 89-97, 158-170 as well as the references cited therein [1,2,3,4].\nMoreover, our experiments support that mitigating large outliers is critical to enable FP8 training.\n- In Figure 3, large activation outliers were present on during cooldown, resulting on divergence of an otherwise stable FP8DPA training run.\n- Figure 5 demonstrates large activation outliers prior to the divergence of an FP8DPA run.\n\nSpeaking about outliers, this brings the topic of outlier amplification profoundly studied on Llama by the prior work [4], introducing the smooth SwiGLU variant. It shows FP8 outliers are amplified by SwiGLU that can act quadratically late in training (cf. Lines 183-191).\nOne of our work's novelties is allowing FP8**DPA** training with **not only** scaled gated activations but a wide range of non-linearities stemming sub-linear in behaviour (GeLU),  Gated (SwiGLU), and especially quadratic activation functions (xIELU).\nThe fact that such a quadratic activation function works smoothly with FOG architecture, as evidenced in the Table addressing the previous point in this rebuttal, although it is expected to amplify outliers, is **an important signal in favour of FOG's robustness**.\nWe aim to make this point, currently raised in Line 183, clearer in the next revision.\n\n[1]: Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, in Advances in Neural Information Processing\nSystems.  \n[2]: Mingjie Sun, Xinlei Chen, J Zico Kolter, Zhuang Liu, 2024, Massive Activations in Large Language Models, in First Conference on Language Modeling.  \n[3]: Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, Thomas Hofmann, 2024, Understanding and Minimising Outlier Features in Neural Network Training, in The Thirty-eighth Annual Conference on Neural Information Processing Systems.  \n[4]: Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry, 2025, Scaling FP8 training to trillion-token LLMs, in The Thirteenth International Conference on Learning Representations."}]

Author rebuttal:

Meta-review: This paper introduces a new architecture that aims to eliminate the outlier and enable the fully FP8 training of the LLM, and achieve the BF16 performance at scale. The author conducted medium-scale experiment with downstream evaluation to show the effectiveness of the proposed architecture. Most reviews agree on the significance of this contribution. However, concerns are raised regarding the insufficient ablation studies, and deeper understandings on why this eliminates the outlier. Nevertheless, this work is a nice contribution to the community and enable future research about low-precision training.
Decision: Accept (poster)