NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs
Authors: Haeun Lee
Omin Kwon
Yeonhong Park
Jae W. Lee
Venue: NeurIPS 2025 poster ()
Keywords: FP8
Quantization
LLM
Dual-Precision
Floating Point Quantization
Primary area: infrastructure

Abstract: Meeting service-level objectives (SLOs) in Large Language Models (LLMs) serving is critical, but managing the high variability in load presents a significant challenge. Recent advancements in FP8 inference, backed by native hardware support, offer a potential solution: executing FP16 models by default, while switching to FP8 models during sudden load surges to achieve higher throughput at the cost of a slight quality degradation. Although this approach facilitates effective SLO management, it introduces additional memory overhead due to storing two versions of the same model. In response, this paper proposes NestedFP, an LLM serving technique that supports both FP16 and FP8 models in a memory-efficient manner by overlaying FP8 parameters onto FP16 parameters, allowing both models to share the same FP16 memory footprint. By leveraging a compact data format for the overlay and a specialized GEMM kernel optimized for this format, NestedFP ensures minimal degradation in both model quality and inference throughput across both FP8 and FP16 modes. NestedFP provides a flexible platform for dynamic, SLO-aware precision selection. The code is available at https://github.com/SNU-ARC/NestedFP.

PestedFP: High-Performance, Memory-Eficient Dual-Precision Floating Point Support for LLMs
Haeun Lee, Omin Kwon, Yeonhong Park", Jae W. Lee"
Department of Computer Science and Engineering Scoul National University
(haeunlee, om@127, ili196 jaewlee]@snu.ce ac.
Abstract
Meeting service-level objectives (SLOs) in Large Language Models (LLMs) serv- ing critical, but managing the high variability in load presents a significant challenge. Recent advancements in FP8 inference, backed by native hardware support, offer a potential solution: executing FP16 models by default, while switch- ing to FP8 models during sudden load surges to achieve higher throughput at the cost slight quality degradation. Although this approach facilitates effec- tive SLO management, it introduces additional memory overhead due storing two versions of the same model. In response, this paper proposes NestedFP, an LLM serving technique that supports both FP16 and FP8 models in memory- efficient manner by overlaying FP8 parameters onto FP16 parameters, allowing both models to share the same FP16 memory footprint. By leveraging compact data format for the overlay and a specialized GEMM kernel optimized for this format, NestedFP ensures minimal degradation in both model quality and infer- ence throughput across both FP8 and FP16 modes. NestedFP provides flexible platform for dynamic. SLO-aware precision selection. The code available https:/// thub. com/ Satm//NNUARC//esteeFF
Introduction
Large Language Models (LLMs) have emerged as foundational components in modern AI systems, enabling a wide range of applications such as virtual assistants, text and code generation, and multi-modal tasks that integrate textual and visual information [1, 7, 21]. Their broad applicability and strong performance have driven widespread adoption, leading to substantial user demand. For example, OpenAI's production-scale models process roughly 100 billion words per day [12] and production query rates frequently surpass 200 requests per second [17, 37J.
To accommodate such large-scale applications, research efforts have focused not only improving serving throughput [16, 38] but also on meeting service-leved objectives (SLOs) [2, 13, 29]. One of the key challenges in achieving SLOs for LLM serving lies in handling dynamic load. Request rates, as well as input and output sequence lengths, can fluctuate significantly over short time intervals. Simply overprovisioning compute resources may mitigate this issue but results in substantial system cost inefficiency.
As promising alternative, explore approach call dual-precision LLM serving, which utilizes FP8 together with FP16, the de facto standard number format for weight parameters. While FP8 may incur slight degradation in model quality, it offers up to 2x higher peak throughput compared to FP16 [22, 27, 28). By operating in FP16 mode under normal conditions and switching to FP8 mode only during sudden load surges, resource overprovisioning This approach achieves a better balance between SLO attainment and service quality than relying solely either FP16 or FP8.
effectively manage dynamic workloads without
"Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Co-deploying models in both formats, however, is non-trivial. Simply keeping both in memory is infeasible due to excessive memory capacity overhead. Storing only FP16 weights and quantizing them to FP8 on the fly when needed results in highly suboptimal FP8 throughput. Therefore, an effective solution is needed to support dual-precision LLM serving without incurring memory overhead or throughput degradation.
In this paper, we present NestedFP, technique that enables efficient dynamic selection between FP8 and FP16 precision from a single unified model representation. NestedFP decomposes each 16-bit weight tensor into two 8-bit components, allowing FP8 inference without additional memory overhead or throughput degradation. Since FP16 mode requires reconstructing 16-bit values by merging the two 8-bit components in this approach, we develop custom General Matrix Multiplication (GEMM) kernel, built on the CUTLASS library [24], that performs accurate on-the- -ly FP16 reconstruction during kernel execution. According to our evaluation, NestedFP enables high-quality, dual-precision LLM serving efficiently: FP8 models generated from decomposed FP16 weights match the quality of existing FPS quantization methods, and our FP16 GEMM kernel incurs only a 6.38% average performance gap relative to the CUTLASS baseline, which translates to 4.98% end-to-end overhead across models.
Our contributions are summarized as follows:
We propose duel-precision LLM serving as an effective strategy to handle dynamic load fluctuations.
We introduce novel data format that supports both high-quality FP16 and FP8 inference in memory-efficien manner.
We develop custom FP16 GEMM kernel for this data format, which performs on-the-fly FP16 value reconstruction during execution.
We demonstrate that NestedFP enables high-quality dual-precision LLM serving with no memory overhead and high throughput, significantly improving SLO attainment compared to FP16-only baselines.
2 Background
2.1 Floating Point Representation
floating point format with & exponent bits and y mantissa bits denoted as ErMy. The exponent width determines the dynamic range of representable values, while the mantissa width governs numerical precision. A floating point value XFP in this format can be expressed as shown in Equation I, where S is the sign bit, Mj are the mantissa bits, the unsigned integer formed by the exponent bits, and is the exponent bias.
(1
X Xpr= (-1)
Σ Mj M3.2 2B-5
E E 1..1...., 1}
(1)
Commonly used 16-bit formats include FP16 (E5M10) and BF16 (E8M7), while prevalent 8-bit formats include E4M3 and E5M2. These formats are natively supported by many modern hardware platforms and have become standard choices in low-precision machine learning computations [14].
2.2 FP8 Quantization for LLMs
Rise of FP8 Quantization for LLMs. While FP16 has become the de facto standard for L.M serving, 8-bit floating-point formats (FP8) are gaining traction for scenarios demanding even greater efficiency. This trend is driven by two main factors: (1) the representational advantages of Hoating-point formats, and (2) the increasing hardware support for FP8 arithmetic.
First, floating-pons point quantization is generally better suited for LLMs than integer quantization due its wider dynamic range. Unlike in computer vision tasks where integer quantization is DovaalentLLMM often exhibit activation outliers amplified by operations such as LayerNorm [28]. These outliers can significantly degrade model performance not properly managed [14]. Although recent work has attempted to mitigate this issue in integer quantization through static outlier-aware techniques [28, 36), these methods still underperform compared floating-poin quantization, which inherently better accommodates long- long-tailed distributions and high dynamic ranges [20, 22, 40].

---

Second, FP8 support is rapidly being adopted by modern hardware accelerators, including NVIDIA Hopper GPUs and Intel Gaudi HPUs [14]. Leveraging this support, FP8 GEMM operations can achieve up to 2x speedup compared to their FP16 counterparts. This growing availability is accelerat- ing the adoption of floating-pont point quantization caantization-pariullllaly FP8 further highlighting its relevance for efficient LLM deployment.
E4M3 Format. Among the two widely adopted FP8 1rmmats-E4M3 and E5M2-exisile studies have consistently shown that E4M3 yields higher inference accuracy for Transformer-basee language models [22, 28, 31]. However, due its reduced dynamic range compared to FP16, the effectiveness of E4M3 critically depends on the application of appropriate scaling strategies [15, 22, 3 31]. Scaling applied align the dynamic range of tensor values with the numerical bounds of the E4M3 format. Scaling factors may be determined statically- based on heuristics such as the absolute maximum, percentile clipping, or mean squared error (MSE) minimization [22]-or or computed dynamically during runtime. Weight tensors are typically scaled statically on a per-channel basis [20. 28, 40], most commonly using the absolute maximum value. Activation tensors, by contrast, may be scaled offline on a per-tensor basis [20, 28, 40] dynamically per-token basis during inference. While the latter approach can improve accuracy, it incurs additional runtime cost due to the overhead of computing scaling factors on-the-fly [14, 27].
FP8 Results. While FP8 inference offers significant throughput gains, comes with the potential caveat of quality degradation due to the inevitable loss of information associated with reduced precision [39]. Table I presents a comparison between FP8 and FP16 across various models and benchmarks. Specifically, for FP8 quantization, we use the E4M3 format with per-token activation and per-channel weight quantization for all linear layers, using the vLLM framework [33]. A consistent accuracy degradation is observed in FP8 models, which clearly shows the trade-off between throughput and model quality in FP8 quantization.
Llama 3.1 8B
Mistral Nemo
Phi-4
Mistral Small
FP16
FP8
FP16
FP8
FP16
FP8
FP16
FP8
Minerva Math
18.2
17.6
-0.60
17.2
16.5
-0.72
42.7
42.9
+0.14
35.4
34.2
-1.14
MMLU Pro
33.3
32.8
-0.49
35.4
35.3
-0.06
53.1
52.7
-0.36
54.3
53.6
-0.77
BBH
39.5
38.6
-0.86
41.6
40.6
-0.97
27.5
27.1
-0.40
52.4
50.8
-1.62
Table 1: Downstream task accuracy for FP8 quantized models relative to FP16 models.
3 Motivation
3.1 Challenge: Dynamic Load in LLM Serving
LLM serving is characterized by substantial variability in request rates, as well as input prompt and output sequence lengths across requests 30J. These variability is particularly problematic because the batch size scheduled at each iteration can fluctuate significantly, directly affecting the large deviations in Time To First Token (TTFT) and Time Per Output Token (TPOT). Such fluctuations occur in a highly dynamic and often unpredictable manner [29], making it difficult for service providers to consistently meet SLOs.
To quantify the extent of this variability, we analyze production traces from Microsoft Azure's LLM inference service [5]. Specifically, we measure how the request rate fluctuates over one-hour interval 100:00-01:00 UTC, May 10, 2024, in the trace). Figure la presents the results. The request rate exhibits nearly fivefold variation between the lowest and highest load periods. Moreover, as shown in the one-minute zoom-in view, these fluctuations occur at high temporal frequency, indicating second-level variability.
classic approach to handling such load variations auto-scaling, which adaptively adjusts com- putational resources (e.g., increasing or decreasing the number of GPUs in a cluster) [8]. However, auto-scaling is primarily effective in cloud environments with abundant resources and typically operates on minute-to-hour timescales [10]. Consequently, it is inadequate for LLM serving systems, which must respond to sharp, second-level load spikes and scale throughput within millisecond-level intervals meet SLO requirements.
3

---

Raw Request Rate (1s)
Average Request Rate
FP16
50
SL0133ms)
M
0.06
Request Rate
40
p90 TPOT (s)
(reas/s)
0.04
20
10
00:00
00:10
00:20
00:30
00:40 00:50
01:00
0.00
10
40
60
Time
Time (s)
(a)
(b)
Figure (a) Request rate fluctuations during the first hour of May 10, 2024, from the Azure LLM inference trace. The inset plot highlights 1-minute variability from the 30-minute mark. (b) Comparison of p90 TPOT across different model precisions, using the Azure LLM inference trace recorded on May 10, 2024.
3.2
Our Proposal: Dual-Precision LLM Serving
To address short-term fluctuations in resource demand during L.LM serving, propose simple yet effective dual-precision strategy that dynamically switches between FP16 and FP8 modes. When the system load is low, FP16 is used to preserve maximum model quality. Under high load conditions, the system falls back to FP8 mode to increase throughput, thereby maintaining responsiveness. Although FP8 mode may incur slight accuracy degradation, is often preferable violating SLOs during periods of request surges.
Figure 1b presents the per-second p90 TPOT under three precision schemes: FP16, FP8, and the proposed dual-precisiog precision format, using the Llama 3.1 8B model on an H100 GPU with vLLM. The request patterns extracted from a 60-second segment of Microsoft Azure's LLM inference trace [5], scaled down to 20% of the original load. The TPOT SL.O threshold is set to 33 ms [6]. As shown, the FP16 model exhibits more frequent TPOT spikes that exceed the TPOT SLO threshold compared to the FP8 model. To further quantify the difference, we analyze the rate of SLO violations across the same trace: 35.8% of requests under FP16 exceeded the threshold, whereas FP8 reduced this to 17.3%. In short, the proposed dual-precision inference can achieve FPS-level SLO compliance while minimizing quality degradation.
3.3 How to Implement Dual-Precision LLM Serving?
To enable dynamic switching between FP16 and FP8 modes in LLM serving, two straightforward implementation strategies can be considered: separate storage and on-the-fly dequantization.
Separate Storage. One simple approach is maintain both FP16 and FP8 models in memory and switch between them according to system load fluctuations. While this enables fast switching, it introduces a 50% memory overhead-a significant burden for LLM serving systems already constrained by model size. This memory capacity overhead not only limits the maximum deployable model size but also reduces the available memory for key-value (KV) caching, thereby restricting the number concurrent requests and ultimately degrading overall serving throughput.
On-the-fly Quantization. An alternative approach is to store only the FP16 model and perform on-the-fly quantization to perform computation in FP8. Unlike the separate storage strategy, this approach avoids additional memory capacity overhead. However, its FP8 performance is suboptimal; although computation carried out by FP8 units with high throughpur, the full 16-bit weights must still be loaded from memory, effectively doubling memory traffic compared to native FP8 exccution. As result, memory bandwidth becomes a bottleneck, limiting the achievable throughput and diminishing much of FP8's advantage over the FP16 baseline.
These limitations highlight the need for more sophisticated dual- precision serving mechanism one that eliminates additional memory overhead while enabling efficient GEMM execution in both FP8 and FP16 modes.

---

PestedFP
Overview. Figure 2 illustrates the core concept of NestedFP, which stores only 16-bit weights in memory and extracts 8-bit weights from them when operating in FP8 mode. Specifically, NestedFP decomposes each FP16 parameter into two parts: the upper 8 bits and the lower 8 bits. With these two parts of the weight parameters stored as separate tensors, both parts are loaded in FP16 mode, whereas only the upper part is loaded in FP8 mode. This approach not only eliminates additional memory usage but also prevents memory traffic amplification for example, fetching 16-bit weights when executing in 8-bit mode.
Challenge I: 8-bit Mode Accuracy. One key challenge of this approach is ensuring that the 8-bit representation extracted from 16-bit parameters achieves sufficient accuracy. In other words, it must maintain performance comparable to existing FP8 quantization methods. Naive truncation (simply using the upper 8 bits of FP16 values) yields representation similar to E5M2, which offers limited precision compared the commonly preferred E4M3 format for LLM inference. Moreover, such truncation often performs worse due to indiscriminate rounding. further degrading accuracy. This motivates the need for a quantization-awe aware data format that enables the extraction of high-quality FP8 representations. Section 4.1 elaborates on this data format design.
Challenge 2: 16-bit Mode Throughput. Another key challenge is maintaining throughput in the 16-bit mode. The 8-bit mode in NestedFP trivially achieves high performance, as native FP8 GEMM kernels can used directly after loading the upper-pede part tensor. However, the 16-bit mode is less straightforward: both the upper and lower-part tensors must be loaded and combined to reconstruct the original 16-bit weights. This requires a custom GEMM kernel capable of performing the reconstruction dynamically. Section 4.2 details this kernel design.
K Wright 8/bit
FP16Activation
K
FP16 Modkd Exrration
NestedEP Upper
8-bit
NestedF PLows
Modkd Excration
Figure 2: FP16 weight is decomposed into 8-bit upper and lower parts. Both parts used for FP16 mode while only upper part is used for FP8 mode.
4.1 Compact yet Effective Data Format for Dual-Precision LLM
Opportunities: Low-Entropy Exponent Bit in FP16 Models. Figure 3a shows the the weight distributions of all FP16 linear layers across four different LLMs. The vast majority of weight values have absolute magnitudes less than or equal 1.75, which means that their most significant exponent bits in FP16 are zero. Specifically, as shown in Figure 3b, three of the four evaluated models exhibit this trait across all layers. The sole exception, Phi-4, contains some layers that deviate from this pattern, but these account for only 8.75% of its layers. This strong tendency suggests an opportunity for lossless exponent mapping from FP16 to the FP8 E4M3 format, which also uses exponent bits.
Probability Density
Llama-3.1-88
Mistral-Small Phi-4
TH #1.75
Mistral-Nemo
Mia
Max.
NewcdHF Applicable Layers
10a
Liase
-0.92
0.91
(1005))
10-2
Nistral Base (128)
0.92
(195
160 (1009)
Phi- (148)
3.50
3.89
146 150 (91.25%)
10-6
Miscral Seall Base (24B)
of 160 (100%))
10c
10-1
10-1
10a
Weight Value (Log Scale)
(a)
(b)
Figure 3: (a) Weight distributions of linear layers across different models. (b) Per-model weight range (min/max) and the proportion of linear layers where NestedFP can be applied.
Our Proposal. Figure 4a illustrates how NestedFP decomposes an FP16 weight such that the upper 8 bits can serve as a high-quality E4M3 representation by leveraging an unused exponent bit in FP1 First, the lower 8 bits directly inherit the lower 8 bits of the FP16 mantissa. The upper 8 bits, which
5

---

correspond to the E4M3 representation, are constructed as follows. The sign bit is directly inherited from FP16. The next four exponent bits are also copied from FP16, excluding the most significant exponent bit, which is assumed to be unused (or zero). Finally, 3 most significant mantissa (i.e., bits M[1:3]) bits are obtained and constructed value applied rounding using a round-to- -oneree-eeeestteeeen policy. Specifically, the 7 least significant bits of the FP16 mantissa (i.e., bits M[4:10] are examined to determine whether the value lies above or below the midpoint (64), If the value is greater than 64, it is rounded up: if it equals 64, it is rounded up only when the least significant bit of the resulting 3-bit mantissa (M3) is 1; otherwise, it is rounded down. This decomposition process is performed offline prior to model execution.
In this scheme, removing most significant exponent bit effectively acts as multiplying 2 between E4M3 and FP16 value. In other words, NestedFP effectively applies E4M3 quantization with fixed scaling factor of 2
FP16 Reconstruction. Figure 4b illustrates how NestedFP reconstructs FP16 weights at runtime. Essentially, this process merges the upper and lower 8-bit parts, but not through a simple concatena- tion. The rounding decision made during decomposition must be reversed by checking an implicit checksum; if the LSB of the upper 8 bits differs from the MSB of the lower 8 bits, a rounding-down correction is applied before recombining the exponent and mantissa bits into a valid FP16 value. This process ensures precise. lossless recovery of the original FP16 weights.
FP16 Weight
NestedEF Upper bit la MM'
NestedFP
M
Mg
M M.
M10
M
M2 reverse round.
E1
Es
F
E
M
M M
Round nearest even
F2
M
S
F
Mr Mg
Mg M Mg
My My MoMM
s E
Eg
M M
M
88
10
NestedFP Upper 8 bit
NestedFP Lower 8 bit
FP16 Weight
(a) Refactoring to NestedFP
(b) Reconstruction of FP16
Figure 4: (a) Decomposition of FP16 weight values into two 8-bit parts (offline). (b) Reconstruction of FP16 weight values (online).
Handling Exception Layers. For layers containing weight values with magnitudes exceeding 1.75 (i.e.. where the MSB of the exponent is possibly non-zero). we retain them in FP16 and always perform FP16 computation. This may reduce the benefits of dual-precision LLM serving, as the throughput advantage of FP8 mode becomes less pronounced. However, it is important to recall that, shown in our analysis of popular models (Figure 3), such layers rare and therefore have minimal impact on the overall effectiveness of the dual -ree-sirn scheme.
4.2 FP16 GEMM Kernel with On-the-fly Reconstruction
Figure illustrates the high-level execution flow of the NestedFP FP16 GEMM kernel compared to standard FP16 GEMM kernel. We assume NVIDIA GPU environment, where the CUTLASS implementation serves as the standard FP16 GEMM kernel, upon which our kernel is also built. the standard FP16 GEMM kernel, weights are loaded into shared memory, transferred to registers, and directly consumed by compute units (e.g., tensor cores in NVIDIA GPUs) without any transformation. In contrast, the FP16 GEMM in NestedFP requires a reconstruction step before tensor core execution; weights are stored as two 8-bit tensors, which must be loaded into shared memory, copied to registers, and then reconstructed into full FP16 values. Only after this reconstruction can the data be used by the tensor cores.
Not to make this additional step of reconstruction introduce a bottleneck, we present two key optimizations: (1) merged bitwise operations and (2) a three-stage pipeline.
Efficient FP16 Reconstruction via Merged Bitwise Operations. FP16 reconstruction involves a large number of 8-bit bitwise operations. To reduce the associated overhead, we fuse four 8-bit operations into a single 32-bit instruction, following the approach in [26, 35]. Algorithm illustrates how FP16 reconstruction is performed using merged bitwise operations. Conceptually, the process iterates over the weight tensor, where each iteration processes four elements. In each iteration, four

---

8-bit values are loaded from the upper and lower tensors (W uugper upper and W Wlower) into 32-bit registers and 1. respectively (Line 5-6). Then. the sign bits are extracted from u and stored another register (Line 7). Next, the algorithm checks whether rounding has occurred by examining (Line 8); if rounding is detected, the effect reversed, thereby recovering the exponent and mantissa parts the upper 8 bits (Line 9). Then, the original upper 8 bits, u_orig, are fully reconstructed by concatenating them with s (Line 10). Finally, u_orig and 1 are fused using NVIDIA's __bytt__perrteeeeeeeeeeee_epeee_ instruction (lines 11-12), completing the reconstruction. Note that all the above operations are simultaneously applied to four 8-bit values through a 32-bit instruction, effectively reducing the total number of bitwise operations required.
Stundard FP16
NestedFFP FP16
Algorithm FP16 Weight Reconstruction
Wagper Wanner upper
5-hit
DRAM
FP16
Upper
Lower
2. W: reconetructed FP16 weight tensor
jotal number elements weight lensor
for O NN stop 4 do
SMEM
FP16 w
FP16
Upper Lower
F
vint32_t Wapper 8 1:1++0
vint32 Wione ex80505c88
uint32_t t
Register
FP16 w
Upper Lower
FP16 W
uint3P_t sub
vint32_t u_orig = (((u aub) > 1) & #x3F3f3FaF)
voorig u_oris
Tensor Core
CUDA Core
Tensor Core
11
Wif i + = byte, perm (uLorig, ex1584)
12
Wii 2: 1 4| betee.permmuooria 0x3726)
Figure 5: FP16 execution flow comparison.
for
Three-Stage Pipeline. The CUTLASS FP16 GEMM kernel, upon which the NestedFP FP16 kemel is built, employs a two-stage pipeline to overlap data transfer and computation. Specifically, the two stages correspond to (1) data movement between shared memory and registers and (2) Tensor Core execution, which are overlapped for high throughput. NestedFP extends this design into a three-stage pipeline to further hide reconstruction overheads. In particular, the three stages in NestedFP are: (1) shared-meemor--to-reisstt data transfer, (2) FP16 reconstruction, and (3) Tensor Core execution.
5
Evaluation
Through extensive experiments, demonstrate that NestedFP is an effective solution for handling fluctuating loads in LLM serving by dynamically switching between FP16 and FP8 modes, as evidenced by the following three key results:
The quality of FP8 model extracted from FP16 parameters in the NestedFP data format matches state-of-the-art FP8 quantization results. (Section 5.1)
The FP16 mode of NestedFP achieves throughput comparable to baseline FP16 inference using the vanilla CUTLASS kernel, despite performing on-the-fly FP16 reconstruction. (Section 5.2 and Section 5.3)
NestedFP significantly improves SLO attainment over FP16-only deployment, without introducing additional memory overhead. (Section 5.4)
5.1
FP8 Model Quality
Methodology. We evaluate the accuracy the FP8 model in NestedFP against FP8 models quantized using commonly adopted configurations, specifically per-channel weight quantization. For activation quantization, we apply per-token quantization for both NestedFP and the baseline. The evaluation covers six models: Llama 3.1 8B, Mistral Nemo (12B), Phi-4 (14B), Mistral Small (24B), Llama 3.1 70B, and DeepSeek- R downstream tasks: Minerva Math, MMLU Pro, and BBH [18, 32, 34]. All models are integrated into the vLLM framework, and accuracy is measured using the LM Evaluation Harness [1 I]] As noted in Section 4. 8.75% of the linear layers in Phi-4 executed in FP16. More details on the experimental setup are provided in Appendix A.
70B [3, 4, 9, 21, 23]. For benchmarks, we use three
applicable to NestedFP and are therefore
Results. Table 2 presents the results. Across all models and benchmarks, the FP8 model in NestedFP achieves accuracy comparable to the baseline FP8 model, demonstrating the effectiveness of the new data format of NestedFP.
7

---

Minerva Math
MMLU Pro
BBH
Model
Size
FP8(B)
FP8(N)
FP8(B)
FP8(N)
FP8(B)
FP8(N)
Llama 3.1
8B
17.6
17.2
32.8
33.2
38.6
39.0
Mistral Nemo
12B
16.5
16.4
35.3
34.7
40.6
40.0
Phi-4
14B
42.9
42.9
52.7
53.3
27.1
28.7
Mistral Small
24B
34.2
35.2
53.6
53.7
50.8
51.1
Llama 3.1
70B
35.7
34.6
48.1
47.4
49.1
49.7
DeepSeek HL--iitilll-Laamm
70B
46.0
45.5
49.2
49.4
53.2
53.4
Table 2: Accuracy of FP8 models in NestedFP across various downstream tasks. FP8(B) denotes the baseline model, and FP8(N) denotes NestedFP.
5.2 FP16 GEMM Kernel Performance
Methodology. We evaluate the FP16 GEMM kernel performance of NestedFP against the CUTLASS FP16 GEMM kernel, upon which ours is built, using an NVIDIA H100 GPU. Note that NestedFP does require any custom kernel for FP8 GEMM; thus, we omit FP8 kernel evaluation. We consider four different weight matrix shapes: (N, K) = (28672, 4096). (28672, 5120), (35840, 5120), (65536, 5120). The (M) dimension, which corresponds the number of tokens processed in parallel, is varied from 32 to 2048 in increments of 32. For both kernels, we select the best configuration (e.g., tile size) for each GEMM shape through an exhaustive search over the design space. Details of the kernel search space are provided in Appendix D.2.
Results. Figure 6 shows the results. The NestedFP kernel consistently demonstrates performance comparable to the CUTLASS baseline across all GEMM shapes. Specifically, the average relative overheads for each weight matrix dimension 5.89%, 6.33%, 4.59%, and 6.33%, respectively. These results indicate that the NestedFP kernel effectively minimizes the overhead introduced by on- the-fly FP16 reconstruction. Additional results for other GEMM shapes, which exhibit similar trends, as well as comparisons with cuBLAS FP16 kernels, are provided in Appendix B and Appendix D.I.
CUTLASS
NestedFP
1000
600
750
Latency (us)
1500
750
400
500
500
1000
200
250
250
500
0
2000
2000
2000
2000
M
M
M
M
28672,
4096
28672, = 5120
N 35840, K 5120
= 65536,
5120
Figure 6: CUTLASS baseline and NestedFP kernel performance comparison.
5.3 FP16 Mode End-to-End Inference Performance
Methodology. We evaluate the end-to-end inference throughput of the FP16 mode of NestedFP by integrating into the vLLM framework on single NVIDIA H100 GPU. Specifically, we set the input and output sequence lengths to 1024 and 512, respectively, while varying the batch size from 32 512. More details on the experimental setup are provided in Appendix C. The results are compared against vanilla vLLM FP16 execution, which relies on PyTorch for invoking GEMM kernels. The FP8-mode throughput of NestedFP excluded from the evaluation, does not require separate kernel.
Results. Figure 7 presents the results. NestedFF achieves FP1 throughput highly comparable to the bascline, with only minor overheads-5.0kk (Llama 3.1 8B), 5.56% (Mistral Nemo), 4.76% (Phi-4), and 4.56% (Mistral Small) on average across batch sizes. The end-to-end throughput degradation is even lower than the kernel-level overheads reported in Section 5.2, since non-GEMM components also contribute to total execution time. Additional end-to-end throughput results are provided in Appendix C.

---

Vanilla vLLM
NestedFP
Throughput (take(s)
6000
4000
3000
2000
8000
2000
2000
2000
1000
32
128
256
512
32
64
128
256
512
128
256
512
32
64
256
5i2
Batch Size
Batch Size
Batch Size
Batch Size
LLaMA3.1(88)
Mistral-Nemo(12B)
Phi-4(148)
Mistral-Small(2245
Figure 7: FP16 inference throughput comparison of four models between vanilla vLLM and NestedFP.
5.4 Impact on SLO Attainment
Methodology. In this section, we present case study demonstrating how NestedFP can improve SLO attainment. We implement simple load-aware mode switching policy for NestedFP that, in each iteration, selects FP8 mode when the number of tokens process exceeds predefined threshold (1024 in this experiment); otherwise, it selects FP16 mode. The evaluation is conducted using Llama 3.1-70B on cluster equipped with four NVIDIA H100 GPUs. As imput, use 1,000 requests sampled from an Azure LLM inference trace while varying the request rate. We measure p90 TTFT, p90 TPOT, and SL.O attainment under two criteria: loose and tight. The loose $L.O is defined as < 10x the average single-request latency, and the tight SLO is defined as & 5x the average single-request latency, following the methodology of a previous study [13].
Results. Table shows the results. NestedFP consistently outperforms the FP16-only baseline across all request rates terms of TTFT, TPOT, and SLO attainment, with particularly pronounced improvements under high request rates. Note that this experiment employs a simple, straightforward switching policy, leaving substantial room for further optimization by integrating NestedFP with more sophisticated policies-such such as those considering KV-cache utilization or incorporating load prediction mechanisms. Nevertheless, these results confirm that NestedFP provides an efficient dual-precision mechanism that can significantly enhance SL.O attainment in LLM serving systems.
Request Rate
Method
Tight SLO (%)
Loose SLO (%)
p90 TTFT
p90 TPOT (ms)
FP16
0.7
14.1
9.9893
162.2
7.47 req/s
NestedFP
72.4
90.0
1.9518
118.9
FP16
71.6
93.6
1.6449
151.3
5.03 req/s
NestedFP
94.6
99.9
0.9679
86.8
FP16
83.1
96.9
1.2404
109.2
3.79 req/s
NestedFP
96.6
100
0.8168
60.1
FP16
90.4
99.4
0.8915
64.9
1.99 req/s
NestedFP
98.4
100
0.6116
47.7
Table 3: SLOO attainment and p90 TTFT/TPOT of Llama-3.1-70B on 4xH100 under varying loads.
Conclusion
We introduce NestedFP, a memory-efficient framework that supports both FP16 and FP8 precisions for LLM inference memory- way. NestedFP introduces new data format that enables direct extraction of FP8 weights from FP16 weights without additional memory overhead. NestedFP also incorporates custom GEMM kernel optimized for this format ensure efficient computation. As result, NestedFP enables dynamic switching between FP16 and FP8 modes with the memory footprint of FP16, without noticeable degradation either accuracy or inference throughput, providing an effective means to improve SL.O attainment in LLM serving.
7
Acknowledgments
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea Government (MSIT) Technology Planning Evaluation (ITPP) under the artificial intelligence semiconductor support program 20--2223-RR-202200056011 funded by the Korea Government (MSIT).
and Institute of Information & Communications

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777 2023.
[2] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gula- vani, Alexey Tumanov, and Ramachandran Ramjec. Taming Shrroughput-Lateneee tradeoff in [LLM) inference with (Sarathi-Serre In 18th USENIX Symposium Operating Systems Design and Implementtionn (OSDI 24), pages 117-134, 2024.
[3] Mistral AL Mistral-neman nemo--emo-aeeeeee faaae-2477,
o- 1-Basee24488 2407, 2024.
[4] Mistral AL. -asstra-small-24--bbe 2abb-base-250.. amall124BB Bas---ase---- 2025.
[5] Microsoft Azure. Azure Ilm inference trace 2024. https://github. com//zzure/AzurePubl 2024.
[6] Baseten. Understanding performance benchmarks for Ilm inference. https: /ww.s//tee/...eeee....m baseten. bo/blog/undeerstndding wernnng--prforrnreeeeeeee weerrmmaneeeeenccaars---ffoe 1i----leemeeeeeee 2025.
[7] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 1877-1901, 2020.
[8] Marta Catillo, Umberto Villano, and Massimiliano Rak. A survey on auto-scaling:: how to exploit cloud elasticity. International Journal of Grid and Utility Computing, 14(1):37-50, 2023.
[9] DeL Deepseck-rl-distill-aammn http::/ilmuaama.iontttttsssgnnnn http:///ccc.ceeepeeeeeaaaaaaaaaa
https://huggingfatt
[10] Javad Dogani, Reza Namvar, and Farshad Khunjush. Auto-scaling techniques in container- based cloud and edge/fog computing: Taxonomy and survey. Computer Communications, 209:120-1500 2023.
[11] ElcutherAI. Am-evaluation-hhre harness. http:///ggtthh.ccoo/eeteeeee///r arness.
[12] Andrew Griffin. Chatgpt creators openai are generating 100 billion words per day, ceo says. https: //www independent... co. uk/ wtwpchhhhhattttpppeaiiiii 0. 0.htm1, 2024.
sam al
[13] Ke Hong, Xiuhong Li, Lufang Chen, Qiuli Mao, Guohao Dai. Xuefei Ning, Shengen Yan, Yun Liang, and Yu Wang. Sola: Optimizing slo attainment for large language model serving with state-aware scheduling, Proceedings of Machine Learning and Systems, 7, 2025.
[14] Jiwoo Kim, Joonhyung Lee, Gunho Park. Byeongwook Kim, Se Jung Kwon, Dongsoo Lee, and Youngjoo Lee. An investigation of fp8 across accelerators for Ilm inference. arXiv e-prints, pages arXiv-2502, 2025.
[15] Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. Fp8 quantization: the power of the exponent. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 14651-14662, 2022.
[16] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611 6-1-626 2023.
10

---

[17] Marina Lammertyn. 60+ chatgpt facts and statistics you need to know in 2025. https:
bbblog.invgate... com/ http//hhttttssttteeee statistics.
[18] Aitor Lewkowyez. Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et Solving quantitative reasoning problems with language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 3843-3 -357,,87,,4 2022.
[19] Jiachen Liu, Jae-Wop Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. Andes: Defining and enhancing arXiv preprint arXiv:2404.16228, 2024.
Ilm-based text streaming services.
[20] Shih- Shih-Yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang-Ting Cheng, Llm-fp4: 4-bit floating- point quantized transformers. In Proceedings of the 2023 Conference Empirical Methods Natural Language Processing, pages 592-605, 2023.
[21] Meta. Introducing Ilama 3.1: :: Our most capable models to date. https: cottt.ooooboogggg /meta-llama- 3-1/, 2024.
[22] Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433, 2022.
[23] Microsoft. Phi-4. httpss//hugginnaaecccccooiiiroro
[24] NVIDIA. Cutlass.
[25] Muhammad Osama, Duane Merrill, Cris Cecka, Michael Garland, and John D Owens. Stream- Work-centric parallel decomposition for dense matrix-matrix multiplication the gpu. In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, pages 429-432 431, 2023.
[26] Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, and Jae W Lee. Any-precision 1lm: low-cost deployment of multiple, different-sized lms.. In Proceedings of the 41st International Conference on Machine Learning, pages 39682-39701, 2024.
[27] Sergio P Perez, Yan Zhang, James Briggs, Charlie Blake, Josh Levy-Kramer, Paul Balanca, Carlo Luschi, Stephen Barlow, and Andrew William Fitzgibbon. Training and inference of large language models using 8-bit floating point. arXiv preprint arXiv:2309.177 17224, 2023.
[28] Haihao Shen, Naveen Mellempudi, Xin He, Qun Gao, Chang Wang, and Mengni Wang. Efficient post-training quantization with fp8 formats. Proceedings of Machine Learning and Systems, 6:483 498, 2024.
[29] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving, In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pages 173-191, 2024.
[30] Ting Sun, Penghan Wang, and Fan Lai. Hygen: Efficient Ilm serving via elastic online-offline request co-location. arXiv preprint arXiv:2501. 14808, 2025.
[31] Xiao Sun, Jungwook Choi, Chia- Yu Chen, Naigang Wang. Swagath Venkataramani, Vijay- alakshmi Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 4900 4909, 2019.
[32] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging binchench tasks and whether chain-of-thouge can solve them. In Findings of the Association for Compu- tational Linguistics: ACL 2023, pages 1300--1305, 2023.
[33] vllm project. vllm. https: //gi com http://o//wmmmoojjee//////////

---

[34] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang. et al. Mmlu-pro: more robust and chal- lenging multi-task language understanding benchmark. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 15266-95290) 2024.
[35] Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, et al. [Quant-LLM}: Accelerating the serving of large language models via IFP6-Centrie Algorithm-System (Co-Design) on modern [GPUs). In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 699-713, 2024.
[36] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pages 38087-38099. PMLR, 2023.
[37] Yuhang Yao, Han Jin, Alay Dilipbhai Shah, Shanshan Han, Zijian Hu, Dimitris Stripelis, Yide Ran, Zhaozhuo Xu, Salman Avestimehr, and Chaoyang He. Scalellm: A resource-frugal Ilm serving framework by optimizing end-to-end efficiency. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 279-289, 2024.
[38] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim. Soojeong Kim, and Byung-Gor Gon Chun. Orca: A distributed serving system for TTransformer-Bassee generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538, 2022.
[39] Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, and Anshumali Shrivastava. 70% size, 100% accuracy: Lossless Ilm compression for efficient gpu inference via dynamic-length float (dfloatl 1). In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
[40] Yijia Zhang, Lingran Zhao, Shijie Cao, Sicheng Zhang, Wengiang Wang, Ting Cao. Fan Yang. Mao Yang, Shanghang Zhang, and Ningyi Xu. Integer or floating point? new outlooks for low-bit quantization on large language models. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have made our contributions visible to abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: As discussed in the main text, performance differences relative to the baseline may fluctuate depending the batch size. To ensure transparency, present performance graphs covering batch sizes the range [32, 2048), with increments of 32.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
This paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have disclosed all the design space configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will provide code along with supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines cnttps://nips.com
blic/guides/Cooo
ssi ssionPoiccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //iis.. httpssccccpbllcccc
ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have disclosed all the design space configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not include statistical significance analysis, it is not the focus of this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have specified that our experiments were conducted using an H100 GPU,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tt./r/neeiisp// http::ppp.ppppp.ttttttcceeeeeeee
Answer: [Yes]
Justification:
We have followed the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we have explained LLM inference serving can be facilitated with our mechanism.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We have no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we have cited all the open source projects we are using.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ con/dataeets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, will provide instructions reproduce experiments and well code in appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
18

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Yes, we have used LLM for writing and visualization and the methods original.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non- non-standard components.
Please refer our LLM policy (https: http::s//////iss...//// // for what should or should not be described.
19

---

A Evaluation Details
Experiments were conducted on NVIDIA H100 GPU with vLLM 0.8.5 (VI engine) and PyTorch 2.7.0+cu12.6. For accuracy assessment, we use the Math Verify metrie on Minerva Math, the Open LLM Leaderboard configuration for MMLU Pro, and a zero-shot setting for BBH. The M dimension is padded to multiples of T. only for kernel microbenchmarks and is disabled in the vLLM integration.
B
Extended Kernel Evaluation
CUTLASS
NestedFP
N=4096, K=4096
N=6144, K=4096
N=28672, K=4096
100
150
Latency (us)
Latency (ws)
(us)
600
400
50
50
200
500
1000
500
1000
1500
2000
1000
1500
2000
Batch Size (M)
Batch Size (M)
Batch Size (M)
N=4096, K=14336
N=5120, K=14336
N=28672, K=5120
300
Latency (μs)
(sn)
750
Latency
200
500
200
100
250
0
0
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Batch Size (M)
Batch Size
Batch Size (M)
N=5120, K=5120
N=5120,
K=17920
7686
K=5120
150
Latency (1s)
200
400
Latency
100
Latency
100
50
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
(M)
Batch Size (M)
(M)
N=35840, K=5120
N=5120,
K=4096
N=5120,
K=32768
1000
1000
Latency (us)
Latency (us)
Latency (us)
500
500
50-
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Batch Size (M)
Batch Size (M)
Batch Size (M)
N=6144, K=5120
N=65536,
5=2020
150
Latency (1s))
1500
Latency
100
500
1000
2000
1500 Batch Size (M)
Batch Size (M)
Figure 8: Comparison of kernel performance between the CUTLASS baseline and NestedFP.
Figure 8 compares NestedFP with the CUTLASS FP16 baseline across 14 (N, K) GEMM shapes for four models: Llama 3.1 8B, Mistral Nemo, Phi-4, and Mistral Small. We vary M from 32 to 2048 in increments of 32. Overall, NestedFP incurs moderate overhead, with an average of 6.38% relative to the FP1 baseline.
20

---

C Extended End-to-End Throughput Evaluation
FP16
NestedFP
Thruughus (toksts)
20000
15000
10000
10000
10000
5000
128
256
512
128
256
512
32
256
512
128
512
Batch Size
Batch Size
Batch
LLaMA3.1(8B)
Mistral-Nemo (12B)
Phi-4(14B)
Mistral-Smail(24B)
(a) (32, 512)
FP16
NestedFP
Throgghput (takels)
6000
4000
3000
2000
4000
2000
2000
1000
128
256
512
0
32
64
128
256
512
64
128
256
512
0
32
64
128
256
512
Batch Size
Batch Size
Batch Size
Batch Size
LLaMA3.1(8B)
Mistral-Nemo(122)
Phi-4(14B)
Mistral-Small(24B)
(b) (1024, 512)
FP16
NestedFP
Throughput (tokso))
7500
7500
4000
5000
5000
5000
2500
2000
32
120
236
512
32
64
120
256
512
32
64
120
256
512
32
64
120
256
512
Batch Size
Batch Size
Batch Size
Batch Size
LLaMA3.1(8B)
Mistral- (12
Phi-4(14B)
Mistral-Smail(244)
(c) (32, 32)
FP16
NestedFP
750
600
1000
Throughput
200
250
200
128
256
512
32
128
256
si2
32
64
120
256
512
32
64
256
5i2
Batch Size
Batch Size
Batch Size
Batch Size
LLaMA3.1(8B)
Mistral-Nemo(12B)
Phi-4(148)
Mistral-Small(24B)
(d) (1024, 32)
Figure 9: FP16 inference throughput comparison across four models under different input/output token configurations.
Figure reports end-to-end throughput for (input, output) token pairs of (32, 512), (1024, 512), (32, 32), and (1024, 32). Across all settings, NestedFP achieves throughput comparable to the FP16 baseline. Chunked prefill was enabled, and the maximum number of batched tokens was set 8192.
21

---

D CUTLASS FP16 GEMM Kernel
D.1 CUTLASS Baseline and Py Torch Kernel Comparison
CUTLASS
PyTorch
N=4096, K=4096
N=6144, K= K=4096
N=28672, K=4096
100
Latency (US)
Latency (Us))
Latency (us)
600
400
200
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Batch Size (M)
Batch Size (M)
Batch Size (M)
N=4096, X=14336
N=5120, K=14336
N=28672, K=5120
Latency (us)
300
Latency (us)
400
750
200
Latency
500
100
250
1000
2000
500
1000
1500
2000
500
1000
1500
2000
Batch Size (M)
Batch Size (M)
Batch Size (M)
N=5120, K=5120
N=5120, K 17920
N 7680, K 5120
Latency (us)
150
Latency (us)
Latency (us)
200
400
100
200
100
0
0
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Batch Size (M)
Batch Size (M)
Batch Size (M)
N-35840, K=5120
N=5120, K=4096
511220 K=32768
1000
1000
Latency (1s)
Latency (us)
100
Latency
500
500
50-
500
1000
1500
2000
500
1000
1500
2000
0
500
1000
1500
2000
Batch (M)
Batch Size (M)
Batch (M)
N=6144, K=5120
N=65536, K= 5=2020
Latency (uS)
Latency (us)
150
1500
100
1000
500
500
1000
1500
2000
500
1000
1500
2000
Batch Size (M)
Batch Size (M)
Figure 10: Comparison of kernel performance between CUTLASS baseline and PyTorch.
Figure 10 compares kernel performance between the CUTLASS FP16 baseline and PyTorch 2.6.0 (CUDA 12.4). We evaluate 14 (N, K) GEMM shapes from four models: Llama 3.1 8B, Mistral Nemo, Phi-4, and Mistral Small, sweeping M from 32 to 2048 in increments of 32. Overall, the CUTLASS bascline achieves performance comparable to Py PyTch.
22

---

D.2 Kernel Search Space
We build upon CUTLASS 3.6.0. In CUTLASS, four warps are grouped into a warp group. We evaluate two variants: won- non-cooperative kernel (one warp group for TMA and one for MMA) and cooperative kernel (one warp group for TMA and two for MMA). For con-cooperative kernels, we grid- grir-search tile dimensions with Tm {16, 32, 64, 128, 256), 7, E {64, 128. 256), and Tą E (64, 128, 256}. For cooperative kernels, we vary only Tw. setting T, E { {128, 256}. We exclude configurations that fail to compile. Non-cooperative kernels use cluster shape of (1, 1, 1), whereas cooperative kernels use (2, 1, 1). We use the persistent scheduler for non-cooperative kernels, and consider both the persistent and Stream-K [25] schedulers for cooperative kernels.
E Extended Applicability Analysis
Model
Layers
# Applicable
Ratio (%)
CodeLlama
224
223
99.6
Cada 13B
280
277
98.9
Gemma 4B
563
429
76.2
Gemma 12B
661
527
79.7
Gemma 27B
759
625
82.3
Llama 3.1 8B
224
224
100.0
Llama 3.1 70B
560
523
93.4
Mistral Nemo 12B
280
280
100.0
Mistral Small 24B
280
280
100.0
Phi-3.5 Mini
128
112
87.5
Phi-4 14B
160
146
91.2
Qwen 3 8B
252
249
98.8
Qwen 3 14B
280
278
99.3
Qwen 3 32B
448
438
97.8
Table 4: Layer-wise applicability of NestedFP across models.
Table 4 presents an upplicability analysis across broader range of models. We report the total number of layers, the number of applicable layers satisfying the weight- hight-mmanitude constraint ([w) S 1.75), and the corresponding ratio. Overall, most models exhibit high applicability across layers.
23


Reviews:
[{"review_id": "KaLOParpd5", "reviewer": "Reviewer_bPHS", "summary": "This paper introduces **NestedFP**, a precision‐adaptive inference system that stores only a standard FP16 checkpoint which can be run entirely in FP8 or lossless FP16 on demand. It works by splitting each FP16 weight into two 8-bit tensors—an “upper” tensor that directly matches the E4M3 FP8 format and a “lower” tensor that, when fused back via bit-wise operations, exactly reconstructs the original FP16 value. A custom, branch-free CUTLASS GEMM kernel overlaps these reconstruction steps with tensor-core math, adding just ~6.5% compute overhead. Unlike prior approaches, there’s no extra memory for scales or metadata, and only a handful of out-of-range “exception” layers fall back to native FP16. Integrated into vLLM, NestedFP delivers up to 1.55× FP8 throughput (≤0.83% accuracy loss) and just 3.9% end-to-end overhead when running in FP16 across 8–24B models on benchmarks like MMLU and BBH.", "questions": "## Questions and Suggestions for the Authors\n\n1. **Precision of NestedFP**  \n   - **Issue:** NestedFP is the same as FP15. In this case, is it necessary for us to use FP16?\n   - **Request:** Please include an analysis or microbenchmarks for a more general model. Do we need 'E1' in the precision? I am not confident here we can save one bit for 'E1'.\n\n2. **Memory Issue Justification**  \n   - **Issue:** The rationale for trading FP8 and FP16’s accuracy/throughput to save memory is not grounded in concrete use cases.  \n   - **Request:** Besides the Azure Worklog, give another case/description of inefficiency of deploying the fp8 and fp16 model at the same time. Also, can the author discuss whether this can be resolved on the scheduling level?\n\n3. **Scalability to Very Large Models**  \n   - **Issue:** Evaluation is limited to small/medium models, leaving its behavior on 10 B+ parameter / Reasoning models unknown.  \n   - **Request:** Provide either (a) a cost projection (memory, compute, communication) for a llama 3.3 - 70B, Qwen-3, or DeepSeek R1.\n\n4. **Detailed Accuracy Comparison**  \n   - **Issue:** The paper reports aggregate metrics but lacks a layer-wise accuracy breakdown against FP16. Also, like DeepSeek FP8 [https://github.com/deepseek-ai/DeepGEMM] has no performance drop if we only apply it on the attention layer, how does NestedFP work in this scenario?\n   - **Request:** Add a table or plot showing per-layer or per-block relative error for NestedFP vs. FP16 vs DeepSeek FP8 on standard benchmarks. \n\n5. **Scheduling Replicate**\n  - **Issue** The paper is motivated by an Azure scheduling scenario, but I didn't see any scheduling analysis.\n  - **Request** Use a simulated/real request flow in the author's vLLM implementation and show the improvement of TTFT / other performance metrics.", "limitations": "Partially yes. The authors need a more thorough discussion of potential scenarios where accuracy loss is acceptable and explicitly evaluate this trade-off. Additionally, elaboration on how NestedFP’s precision compromise affects specific critical reasoning would enhance the paper’s impact. There is not potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n### Quality\n- **Practical Implementation and Solid Evaluation** The author has a super solid implementation on vLLM and A branch-free CUTLASS GEMM kernel hides the bit-manipulation overhead (< 6.5 %). \n\n### Clarity\n- **Illustrative figures.** Diagrams of the dual-tensor layout and pipelined kernel aid understanding of low-level mechanics.\n\n### Significance\n- **Per-iteration precision switching.** Enables dynamic adjustment to workload between FP16 and FP8.\n\n### Originality\n- **Novel dual-8-bit decomposition.** Re-purposes the unused FP16 exponent bit to encode an FP8 tensor, yet still reconstructs exact FP16 values.  \n\n---\n\n## Weaknesses\n\n### Quality\n- **Weight distributions requirement** The whole idea is built on the motivation that the majority of weight values have absolute magnitudes less than or equal to 1.75. If this condition not exist, the nestedfp cannot be achieved.\n\n### Clarity\n- **Scattered narrative.** The introduction over-emphasizes FP8 background, delaying presentation of the core mechanism.  \n- **Confusing Motivation** It needs a lot of effort to understand Figure 1 (a). The author can show the figure in 0/1 => under / above the average usage.\n\n### Significance\n- **Exception-layer overhead.** “Few” fallback layers are reported but not analyzed per model; impact on larger models is unclear to me.  \n- **Necessity analysis.** Azure Worklog is not enough for me. I want to see if this issue exists in the LLM production environment.\n- **NestedFP vs FP16** If we use this way to restore fp16 back, the weakness of NestedFP will bring performance and accuracy drop.\n\n### Originality\n- **Literature contrast.** Similar on-the-fly quant-dequant methods exist; a deeper comparison to dynamic quantization kernels is needed.", "was_revised": true, "final_justification": "The technical concerns I had with this paper are mostly resolved. I will raise my point.\n\nPlease, include make sure the discussion in the final manuscript.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. \n&nbsp;  \n1. **Precision of NestedFP** FP16 is a standard data type widely supported by hardware accelerators, including tensor cores. While it is possible to store weights using a 15-bit representation (e.g., by omitting one exponent bit as in FP15), doing so requires custom kernels to reconstruct valid FP16 values at runtime—just as NestedFP does. The key insight is that we're not creating FP15 for its own sake, but rather enabling dual-precision execution from a single representation. To evaluate the general applicability of NestedFP, we provide a detailed analysis of 14 models in Appendix F. These results suggest that a wide range of state-of-the-art models can benefit from NestedFP.  \n&nbsp;  \n2. **Memory Issue Justification:Why We Should Save Memory** \nFirst and foremost, deploying FP8 and FP16 models separately increases memory usage by approximately 1.5× compared to deploying a single FP16 model, which can limit the maximum model size that fits within GPU memory.  \n&nbsp;  \nSecond, in LLM inference, memory capacity is not merely a resource constraint—it has direct implications on performance. Memory savings can often be reinvested to improve system efficiency. For example, memory savings may enable larger batch sizes—thus increasing throughput—or reduce the frequency of KV cache evictions, which can become a significant bottleneck under memory pressure. While we did not incorporate such reinvestment scenarios in our evaluation in order to isolate the performance impact of kernel slowdown, we believe that, when leveraged, the performance gains enabled by NestedFP’s reduced memory footprint can more than offset the throughput loss caused by kernel slowdown (i.e., 3.9%).  \n&nbsp;  \nTo evaluate how memory savings of NestedFP can be invested to performance improvement, we ran controlled experiments using an H100 GPU with Azure traces (791 requests, 5 minutes) and compared NestedFP16 only deployment (no weight duplication, full KV cache capacity) with a dual deployment of FP16 and FP8 models (weight duplication, KV cache capacity reduced from 34GB → 12GB).  Severe performance degradation was observed when co-deploying both models. P90 TTFT increased from 4.02s → 283.31s, total E2E latency more than doubled (from 300s → 627s), and throughput dropped by > 40%. This was mainly due to limited batch sizes and frequent KV cache evictions.  \n&nbsp;  \nIn short, the modest kernel slowdown is a worthwhile trade-off, as the resulting memory savings can be reinvested to mitigate key performance bottlenecks, ultimately outweighing the overhead.  \n&nbsp;  \n3. **Evaluation on Large Models/Reasoning Models** We evaluated the effectiveness of NestedFP on two 70B-scale models: LLaMA 3.1 70B, a general-purpose language model, and DeepSeek-R1-Distill-LLaMA-70B, which is specialized for reasoning tasks.  \n&nbsp;  \n- *Applicability*: We analyzed the weight distributions of LLaMA 3.1 70B and DeepSeek-R1-Distill-70B to assess the applicability of NestedFP across QKVO and MLP linear layers. For both LLaMA 3.1 70B and DeepSeek-R1-Distill-70B, over 93% of layers (523/560 and 521/560, respectively) satisfied the constraint. While the remaining layers are left in FP16, most layers can be converted to the NestedFP format. This demonstrates that NestedFP is applicable to both large-scale models and reasoning models.  \n- *Accuracy*: We benchmarked both models on three tasks: Minerva MATH, BBH, and MMLU Pro. We compared FP16 results, FP8 baseline (per-token activation and per-tensor weight quantization), and our NestedFP8 format. Results on the table show that NestedFP8 consistently matches or outperforms the FP8 baseline across benchmarks, while maintaining accuracy very close to full-precision FP16. These results confirm that NestedFP8 is effective for both large-scale and reasoning models.  \n\n| Model | Benchmark | FP16 | FP8 Baseline | NestedFP8 |\n|-------|-----------|------|-------------|-----------|\n| LLaMA 3.1 70B | Minerva | 0.3522 | 0.3526 | 0.3546 |\n|              | BBH     | 0.4956 | 0.4942 | 0.4933 |\n|              | MMLU Pro | 0.4751 | 0.4791 | 0.4745 |\n| DeepSeek-R1-Distill-70B | Minerva | 0.4534 | 0.4508 | 0.4468 |\n|                         | BBH     | 0.5365 | 0.5308 | 0.5356 |\n|                         | MMLU Pro | 0.4960 | 0.4897 | 0.4934 |\n- *Latency*: \nWe measured inference latency on LLaMA 70B using H100 GPUs with tensor parallelism (TP=2). NestedFP8 achieves average x1.43 TTFT speedup over FP16 across different input lengths, while NestedFP16 incurs minimal overhead (<4% increase). These results confirm that the NestedFP format overhead remains negligible at large scale, while FP8 quantization provides significant speedup as GEMM operations dominate the computation.\n\n| Tokens (In, Out) | Metric | FP16  | FP8   | NESTEDFP16 | NESTEDFP8 | \n|------------------|--------|-------|-------|------------|-----------|\n| (1024, 32) | TTFT(ms) | 161.9 | 119.9 | 167.7 | 114.4 |\n|            | TPOT(ms) | 29.5  | 20.5  | 30.4  | 21.5  |\n| (2048, 32) | TTFT(ms) | 302.1 | 203.1 | 315.9 | 207.0 |\n|            | TPOT(ms) | 29.6  | 20.7  | 30.5  | 21.7  |\n\n4. **Detailed Accuracy Comparison** To perform a per-layer analysis of the decoder, we quantized all QKVO and MLP FP16 linear layers into FP8 and measured the relative error across decoder layers using the Pile training dataset. Results for selected layers (L10, L20, L30, L40) and the full-model average show that NestedFP8 generally yields slightly higher error than DeepSeek FP8 due to its coarser quantization. As NestedFP8 applies only to linear layers, attention-layer quantization schemes like DeepSeek FP8 can still be applied independently without conflict.\n\n| Model          | Method        | L10    | L20    | L30    | L40    | Avg.   |\n|:--------------|:--------------|:------:|:------:|:------:|:------:|:-----:|\n| Llama 3.1 8B   | DeepSeek FP8  | 0.088  | 0.075  | 0.080  |   -    | 0.077 |\n|                | NestedFP8     | 0.112  | 0.090  | 0.096  |   -    | 0.095 |\n| Mistral Nemo   | DeepSeek FP8  | 0.079  | 0.081  | 0.075  | 0.084  | 0.076 |\n|                | NestedFP8     | 0.097  | 0.097  | 0.089  | 0.099  | 0.094 |\n| Phi-4          | DeepSeek FP8  | 0.056  | 0.072  | 0.058  | 0.063  | 0.059 |\n|                | NestedFP8     | 0.064  | 0.082  | 0.066  | 0.069  | 0.066 |\n| Mistral Small  | DeepSeek FP8  | 0.055  | 0.070  | 0.062  | 0.067  | 0.060 |\n|                | NestedFP8     | 0.072  | 0.086  | 0.077  | 0.092  | 0.077 |\n\n5. **Scheduling Replicate** \nFirst and foremost, we would like to emphasize that the primary goal of our work is to enable per-iteration precision switching (every 10–100 ms) between FP8 and FP16 without model duplication. NestedFP is scheduler-agnostic and can be seamlessly integrated with any LLM scheduling policy. While the design of such schedulers is beyond the scope of this paper, we consider developing effective scheduling strategies for NestedFP a promising direction for future research.  \n&nbsp;  \nNevertheless, to provide a rough estimate of the potential performance benefits in a realistic serving scenario, we implemented a simple load-aware scheduler on the vLLM v1 engine. This scheduler selects NestedFP8 when the batch token count exceeds half of max_num_batched_tokens, and defaults to NestedFP16 otherwise—achieving adaptive precision switching without forecasting or user input. Note that this is a very naive policy, and many optimization opportunities remain on the table, such as incorporating per-query SLO constraints, KV cache utilization, or more sophisticated load prediction mechanisms.  \n&nbsp;  \nWe conducted experiments on a single-node H100 80GB server using Mistral Small 24B as the test model Azure LLM trace. In addition to TTFT and TPOT, we evaluated SLO attainment by defining two SLO classes following the latency analysis framework suggested by SOLA (MLSys'25). Specifically, we set tight SLO as ≤ 5× the average single-request latency and loose SLO as ≤ 10× the average single-request latency, where the single-request latency was measured in our experiment environments.  \n&nbsp;  \nEven with simple policy, being able to dynamically switch between FP8 and FP16 led to dramatic latency improvements over static FP16-only serving:  **p90 TTFT decreased from 2.42s → 0.89s (x0.36), p90 TPOT decreased from 0.26s → 0.063s (x0.24), tight SLO attainment increased from 49.9% → 95.1% (x1.90) and loose SLO attainment increased from 89.5% → 100% (x1.12).**  \n&nbsp;  \nThese results demonstrate the effectiveness of token-level precision switching, highlighting the motivation behind NestedFP, which is to enable it in a memory-efficient manner."}, {"review_id": "A4iyPN7Jzs", "reviewer": "Reviewer_R4or", "summary": "NestedFP is a precision-adaptive weight format that splits every FP16 weight into two 8-bit halves. The upper half alone forms an E4M3-compatible tensor for fast FP8 inference, while both halves can be fused on-the-fly—via a custom CUTLASS GEMM kernel—to recover the exact FP16 value for accuracy-critical execution. Integrated into vLLM, the authors report up to 1.55× higher throughput in FP8 mode with negligible accuracy loss on Minerva-Math, MMLU-Pro and BBH, and only around 4% average overhead when running in FP16 mode, all without duplicating model weights in memory.", "questions": "- When reading only the upper 8 bits for FP8 inference, does it involve scattered loads that incur extra L1/L2 misses versus a contiguous FP8 checkpoint? Any empirical latency numbers compared with native FP8 tensors?\n- Could NestedFP (or its variants) be generated directly from BF16?\n- Given the next-gen Blackwell, can the same “nested” trick be extended to 4-bit? (I assume it's hard to directly transform it to MXFP4 or NVFP4; just want to know some insights.)\n- The throughput report shows that NestedFP8 gains shrink with a large batch compared to vanilla FP8. Is this due to doubling weight-memory traffic (16B) or to reconstruction pipeline saturation?\n- Could you report latency or TTFT.\n- DFloat11 (https://arxiv.org/abs/2504.11651) is related, I'd love to see some comparison if possible.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This method proposes an elegant nested representation that re-uses the unused FP16 MSB to embed an FP8 mantissa, enabling loss-free round-trips while keeping the checkpoint size unchanged.\n- Thoughtful kernel design (SIMT fusion, three-stage pipeline) hides most reconstruction latency.\n- System-level integration inside vLLM with comprehensive and realistic evaluation, and also comprehensive ablations on kernel.\n- Zero extra RAM compared with pure FP16.\n\nWeaknesses:\n- Evaluation coverage, it would be better to have:\n  - larger-scale models that are common in production\n  - Reasoning-oriented checkpoints such as DeepSeek-R1, DS-Distiled-Llama, etc.\n  - Datasets of coding or those that need longer generation, like LiveCodeBench\n- Performance compromise: several datasets still show non-trivial accuracy degradation and, the throughput occasionally falls below vanilla ones\n- A principled treatment of extremely large weights is still missing, limiting universality.\n- Many open-weights today are stored in BF16. Converting BF16 → NestedFP adds an extra cast step—yet the paper does not evaluate this path.", "was_revised": true, "final_justification": "NestedFP is an interesting work that splits every FP16 weight into two 8-bit halves and switch the floating point format on-the-fly. During rebuttal, the authors answered my questions and overall, this is a solid work with both good motivation and implementation.", "rebuttal": "We sincerely thank the reviewer for the detailed and insightful feedback.\n\n1. **Evaluation on Large Models/Reasoning Models** We evaluated the effectiveness of NestedFP on two 70B-scale models: LLaMA 3.1 70B, a general-purpose language model, and DeepSeek-R1-Distill-LLaMA-70B, which is specialized for reasoning tasks.  \n&nbsp;  \n- *Applicability*: We analyzed the weight distributions of LLaMA 3.1 70B and DeepSeek-R1-Distill-70B to assess the applicability of NestedFP across QKVO and MLP linear layers. For both LLaMA 3.1 70B and DeepSeek-R1-Distill-70B, over 93% of layers (523/560 and 521/560, respectively) satisfied the constraint. While the remaining layers are left in FP16, most layers can be converted to the NestedFP format. This demonstrates that NestedFP is applicable to both large-scale models and reasoning models.  \n- *Accuracy*: We benchmarked both models on three tasks: Minerva MATH, BBH, and MMLU Pro. We compared FP16 results, FP8 baseline (per-token activation and per-tensor weight quantization), and our NestedFP8 format. Results on the table show that NestedFP8 consistently matches or outperforms the FP8 baseline across benchmarks, while maintaining accuracy very close to full-precision FP16. These results confirm that NestedFP8 is effective for both large-scale and reasoning models.  \n\n| Model | Benchmark | FP16 | FP8 Baseline | NestedFP8 |\n|-------|-----------|------|-------------|-----------|\n| LLaMA 3.1 70B | Minerva | 0.3522 | 0.3526 | 0.3546 |\n|              | BBH     | 0.4956 | 0.4942 | 0.4933 |\n|              | MMLU Pro | 0.4751 | 0.4791 | 0.4745 |\n| DeepSeek-R1-Distill-70B | Minerva | 0.4534 | 0.4508 | 0.4468 |\n|                         | BBH     | 0.5365 | 0.5308 | 0.5356 |\n|                         | MMLU Pro | 0.4960 | 0.4897 | 0.4934 |\n- *Latency*: \nWe measured inference latency on LLaMA 70B using H100 GPUs with tensor parallelism (TP=2). NestedFP8 achieves average x1.43 TTFT speedup over FP16 across different input lengths, while NestedFP16 incurs minimal overhead (<4% increase). These results confirm that the NestedFP format overhead remains negligible at large scale, while FP8 quantization provides significant speedup as GEMM operations dominate the computation.\n\n| Tokens (In, Out) | Metric | FP16  | FP8   | NestedFP16 | NestedFP8 | \n|------------------|--------|-------|-------|------------|-----------|\n| (1024, 32) | TTFT(ms) | 161.9 | 119.9 | 167.7 | 114.4 |\n|            | TPOT(ms) | 29.5  | 20.5  | 30.4  | 21.5  |\n| (2048, 32) | TTFT(ms) | 302.1 | 203.1 | 315.9 | 207.0 |\n|            | TPOT(ms) | 29.6  | 20.7  | 30.5  | 21.7  |\n\n2. **Regarding the question on reading FP8 tensors in inference** In our implementation, the two 8-bit tensors (upper and lower) are stored as separate, fully contiguous FP8 checkpoints, each aligned in memory to prevent scattered access. During FP8 inference, only the upper tensor is loaded, and since it is accessed contiguously, it incurs no additional L1/L2 cache misses compared to native FP8 checkpoints.  \n&nbsp;  \n3. **How to support BF16 weights?** If a model is released in BF16, it can be cast to FP16 during preprocessing and subsequently converted into NestedFPweights. The error introduced by this casting is usually negligible in practice. To demonstrate it, we conducted an element-wise comparison between the FP16-cast versions of LLaMA 3.1 8B and Qwen3 14B and their original BF16 counterparts and found out that not a single parameter in any layer showed a numerically meaningful deviation (≥ 1e-10 absolute or relative error). This negligible deviation is also reflected in the perplexity evaluations shown in the table below. Across all cases, casting to FP16 has minimal impact, supporting the use of FP16 as a preprocessing step for NestedFP.\n\n| Model          | Wikitext2 (BF16) | Wikitext2 (FP16) | C4 (BF16) | C4 (FP16) | PTB (BF16) | PTB (FP16) |\n|----------------|------------------|------------------|-----------|----------|-------------|------------|\n| Llama-3.1-8B   | 6.24             | 6.24             | 9.59      | 9.58     | 9.02        | 9.01       |\n| Mistral Nemo   | 5.75             | 5.74             | 9.58      | 9.57     | 8.66        | 8.66       |\n| Phi-4          | 6.46             | 6.46             | 12.07     | 12.07    | 9.32        | 9.32       |\n| Mistral Small  | 5.29             | 5.29             | 9.09      | 9.09     | 8.00        | 8.00       |\n  \n4. **Throughout reduction of NestedFP8 compared to Torch FP8**\nWe used a custom CUTLASS kernel for FP8 evaluation. The observed performance gap is likely due to suboptimal kernel configuration (e.g., tile shapes), as we did not perform an extensive kernel tuning process—FP8 kernel optimization was not the primary focus of this work. However, based on your comment, we revisited this issue and found that there is no reason to use a custom kernel for NestedFP8; the same kernel used in the baseline can be applied. We will update our evaluation accordingly in future revisions and expect no  performance gap for FP8 GEMM.  \n&nbsp;  \n5. **Latency (TTFT and TPOT)**  \nWe implemented a simple load-aware scheduler on the vLLM v1 engine. This scheduler selects NestedFP8 when the batch token count exceeds half of max_num_batched_tokens, and defaults to NestedFP16 otherwise—achieving adaptive precision switching without forecasting or user input. We conducted experiments on a single-node H100 80GB server using Mistral Small 24B as the test model Azure LLM trace.  \n\n| Metrics       | FP16 | NestedFP |   |   |   |   |   |   |   |\n|---------------|------|----------|---|---|---|---|---|---|---|\n| p90 TTFT (ms) | 2420 | 890      |   |   |   |   |   |   |   |\n| p90 TPOT (ms) | 260  | 63       |   |   |   |   |   |   |   |\n\n6. **Regarding DFloat11** DFloat11 leverages unused exponent bits in bfloat16 and introduces an efficient inference kernel. While both DFloat11 and NestedFP exploit unused exponent bits and employ custom kernels, their goals and effects differ. DFloat11 focuses on compressing model weights and using the saved memory to achieve performance gains, particularly in comparison to CPU offloading scenarios, without any loss in accuracy. In contrast, NestedFP does not reduce the actual model size. Instead, it supports both lossless FP16 inference and quantized FP8 inference to better meet SLO constraints. As mentioned earlier, even with a simple scheduler, NestedFP can deliver substantial latency improvements over static FP16-only serving.  \n&nbsp;  \n7. **Regarding possible extensions to FP4** We appreciate the suggestions regarding both FP4 extensions. Since (1) FP4 requires more careful handling of scaling, and (2) FP8 offers very few unused exponent bits to exploit, extending similar ideas to FP4 is non-trivial—but still a direction worth exploring."}, {"review_id": "ooM7EFUIjS", "reviewer": "Reviewer_CdLp", "summary": "The authors propose a method, NestedFP, to store both FP8 and FP16 in one 16-bit storage, which enables more memory efficient LLM serving compared to storing both models individually and more runtime-efficient than on-the-fly quantization. They provide a custom GEMM implementation built on CUTLASS.", "questions": "Why not show Table 2 as differences, as you did before for FP8 vs FP16 in Table 1?", "limitations": "could be made more explicit into a small section. But they are detailed in the Checklist.\n\nI am very much not an expert in this topic, so while this looks all fine to me, I can not actually gauge the work to its full depth.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- reduced memory requirement\n- good runtime speed-ups in FP8 with small cost in FP16\n- very straightforward idea, and efficient implementation\n- good experiments across multiple models to validate the general applicability\n    - It wouldn’t hurt to go a little broader with the benchmarks though\n\nWeaknesses\n\n- 6.5% increase in GEMM runtime when using FP16\n- This paper feels like it should go into a venue like USENIX ATC/OSDI or so. It does not really strike me as AI research. (Ofc, I will review under the assumption that NeurIPS is in fact the right venue)\n- There is nothing really wrong with this paper, but I cannot gauge the impact. The method seems easy enough to adopt, but there are some significant tricks that had to be made to get the GEMM to be efficient, which sound like they could be overthrown by the next hardware change. And I wonder what experts think of the tradeoff between 3.9% increase end to end in FP16 for the benefit of less memory.", "was_revised": true, "final_justification": "Thank you for the clarifications.\nThe paper overall seems fine to me, but I am not confident enough to give it a higher score. I will maintain.", "rebuttal": "Thank you for raising this thoughtful and important concern. \n\n1. **Tradeoff Between Performance Overheads and Memory Savings** \nIt is true that NestedFP incurs a small performance overhead (e.g., 3.9% throughput loss) in exchange for memory savings. In response to whether this tradeoff is worthwhile, we emphasize that in LLM inference, memory savings can often be reinvested to improve overall performance. For example, memory savings may enable larger batch sizes—thus increasing throughput—or reduce the frequency of KV cache evictions, which can become a significant bottleneck under memory pressure. While we did not incorporate such reinvestment scenarios in our evaluation in order to isolate the performance impact of kernel slowdown, we believe that, when leveraged, the performance gains enabled by NestedFP’s reduced memory footprint can more than offset the 3.9% throughput loss.  \n&nbsp;  \nTo evaluate how memory savings of NestedFP can be invested to performance improvement, we ran controlled experiments using an H100 GPU with Azure traces (791 requests, 5 minutes) and compared NestedFP16 only deployment (no weight duplication, full KV cache capacity) with a dual deployment of FP16 and FP8 models (weight duplication, KV cache capacity reduced from 34GB → 12GB).   \n&nbsp;  \nSevere performance degradation was observed when co-deploying both models. P90 TTFT increased from 4.02s → 283.31s, total E2E latency more than doubled (from 300s → 627s), and throughput dropped by > 40%. This was mainly due to the limited batch size and frequent kv cache evictions.\n&nbsp;  \nIn short, the modest kernel slowdown is a worthwhile trade-off, as the resulting memory savings can be reinvested to mitigate key performance bottlenecks, ultimately outweighing the overhead.  \n&nbsp;  \n2. **Regarding Presentation Clarity and Table Formatting** Thank you for your overall positive comments and thoughtful suggestions. As you pointed out, showing differences in Table 2—similar to Table 1—would improve clarity. We agree and will revise the table to explicitly highlight the differences in FP8 results as well."}, {"review_id": "VeZzANuek3", "reviewer": "Reviewer_B4rE", "summary": "The paper presents NestedFP, a dual-precision weight representation scheme that enables both FP16 and FP8 execution from a single 16-bit model representation, avoiding additional memory overhead. It proposes a structured decomposition of FP16 weights into two 8-bit components, allowing FP8 inference with high throughput and FP16-level accuracy when needed. A custom CUTLASS-based FP16 GEMM kernel is developed to reconstruct full-precision weights on-the-fly. Integration with the vLLM serving framework demonstrates up to 1.55× throughput improvements in FP8 mode and minimal overhead (~3.9%) in FP16 mode. The paper provides a new perspective for LLM serving with dynamic precision adaptation based on system load and service-level objectives (SLOs).", "questions": "1. The proposed method sets a fixed scaling factor for FP8 quantization. Will this limit the flexibility of FP8 quantization? Table 2 partly answers this question. But do you have more ablations for this design – particularly the PPL (perplexity) of the LLM?\n\n2. What are the FP16 accuracy / PPL numbers in table 2?\n\n3. What is the baseline FP16 latency in Figure 7 (b)?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n1. The paper presents a new perspective for LLM serving with dynamic precision adaptation based on system load and service-level objectives (SLOs).\n2. The paper provides evaluation across different LLMs such as LLaMA 3.1 8B, Mistral Nemo, Phi-4, and Mistral Small. It also introduces system-level optimizations such as CUTLASS-based GEMM kernel, SIMT pack instructions, and integration to serving framework vLLM.\n\n\n[Weaknesses]\n1. The motivation of the proposed method is not strong enough. The FP8 quantization already achieves very decent accuracy for LLMs, particularly with post-training quantization techniques such as SmoothQuant and even QAT (quantization-aware training) methods. The motivation to switch between FP8 and FP16 precision is not convincing enough.\n2. Some of the claims/experiments in the paper need further clarification (see questions).", "was_revised": true, "final_justification": "The authors' rebuttal has addressed most of my concerns. And I have raised my score accordingly.", "rebuttal": "Thank you for your thoughtful review and helpful questions. \n\n1. **Doesn’t the SOTA FP8 model work as well as the FP16 model? Why keep both?**\nIt is true that state-of-the-art methods such as SmoothQuant have advanced FP8 quantization to the point where it appears nearly lossless on some popular benchmarks. However, recent studies have shown that, when evaluated more thoroughly across a broader range of tasks, even the best-performing 8-bit models still significantly underperform compared to their FP16 counterparts [1]. As an example, human evaluation results from LLM Arena [2] highlight that the 8-bit version of Llama-3.1-405B-Instruct (FP8) exhibits noticeable performance degradation compared to its FP16 counterpart, particularly in tasks involving coding (score: 1277 vs. 1293) and long-context queries (1275 vs. 1282). Note, that Llama-3.1-405B-Instruct (FP8) used SmoothQuant for FP8 quantization [3]. This highlights the need to retain FP16 alongside FP8 to balance inference quality with SLO constraints, as FP8 may still introduce non-negligible errors in some cases.  \n&nbsp;  \n[1] Tianyi Zhang, Yang Sui, Shaochen Zhong, Vipin Chaudhary, Xia Hu, and Anshumali Shrivastava. 70% size, 100% accuracy: Lossless llm compression for efficient gpu inference via dynamic-length float, 2025.  \n[2] https://x.com/lmarena_ai/status/1835760196758728898  \n[3] https://arxiv.org/pdf/2407.21783  \n&nbsp;  \n2. **PPL Evaluations**\nWe report perplexity results on Wikitext2 (W2), C4, and PTB datasets across four models. The FP8 baseline—denoted as (B)—applies per-token activation and per-tensor weight quantization. Our method, NestedFP, is denoted as (N). Additionally, we report results for SmoothQuant, denoted as (S), which builds on (B) with additional activation smoothing.  \n&nbsp;  \nAll quantization schemes were applied to the QKVO and MLP linear layers. For SmoothQuant, we followed the protocol from the original paper and computed activation statistics using the Pile training dataset. A grid search over α ∈ {0.5, 0.7, 0.9} was performed, and the best perplexity value was selected.  \n&nbsp;  \nOverall, FP8 quantization methods yield slightly higher perplexity than FP16, while SmoothQuant shows comparable performance to the FP8 baseline. The perplexity trends are similar with accuracy evaluations.  \n&nbsp;  \n| Model            | W2(FP16) | W2(B) | W2(N) | W2(S) | C4(FP16) | C4(B) | C4(N) | C4(S) | PTB(FP16) | PTB(B) | PTB(N) | PTB(S) |\n|------------------|-----------|--------|--------|--------|-----------|--------|--------|--------|-------------|---------|---------|---------|\n| Llama 3.1 8B     | 6.24      | 6.30   | 6.31   | 6.29   | 9.58      | 9.67   | 9.67   | 9.67   | 9.01        | 9.06    | 9.08    | 9.06    |\n| Mistral Nemo     | 5.74      | 5.80   | 5.81   | 5.78   | 9.57      | 9.65   | 9.67   | 9.63   | 8.66        | 8.73    | 8.75    | 8.70    |\n| Phi-4            | 6.46      | 6.47   | 6.49   | 6.48   | 12.07     | 12.09  | 12.11  | 12.10  | 9.32        | 9.34    | 9.35    | 9.34    |\n| Mistral Small    | 5.29      | 5.35   | 5.35   | 5.34   | 9.09      | 9.20   | 9.21   | 9.20   | 8.00        | 8.07    | 8.07    | 8.06    |\n&nbsp;  \n3. **FP16 results for models/tasks in Table 2** \nFP16 results are provided in Table 1. To improve clarity, we will include them in Table 2 in the future revision.  \n&nbsp;  \n4. **Regarding Baseline FP16 Latency of Figure 7 (b)** Please refer to Appendix C for the FP16 baseline latency values. In Figure 7(b), “Lvl 3” corresponds to our optimized NestedFP16 kernel. Appendix C also includes a comparison graph for the GEMM shape M×5120×32768, illustrating the performance of both the FP16 baseline and NestedFP16.  \n&nbsp;  \n5. **Regarding scaling factor** The reviewer is correct that NestedFP uses a fixed scaling factor when encoding the FP16 weight tensor into two FP8 components. This design choice enables exact reconstruction of the original FP16 weights, while maintaining compatibility with FP8 tensor operations.  \n&nbsp;  \nImportantly, this does not preclude additional scaling: weights may still be scaled as long as the values remain within the representable range of FP8 (E4M3), offering flexibility where needed.  \n&nbsp;  \nCrucially, our method places no restriction on how activations are quantized. While we adopt per-tensor activation quantization in our experiments, per-token quantization schemes—like those used in the FP8 baseline (B)—can be readily incorporated to further improve accuracy. In this sense, while weight quantization is fixed for exact reconstruction, activation quantization remains fully flexible.  \n&nbsp;"}]

Author rebuttal:

Meta-review: The paper introduces NestedFP, a dual-precision weight representation that encodes each FP16 weight as two 8-bit parts so that a single 16-bit checkpoint can execute either as fast FP8 (using only the upper 8-bit tensor) or as exact FP16 (by fusing both parts on the fly). A custom, branch-free CUTLASS GEMM overlaps reconstruction with tensor-core compute and is integrated into vLLM. Reported results show up to 1.55× higher throughput in FP8 mode with negligible accuracy loss on reasoning benchmarks, and ~3.9–4.5% average end-to-end overhead in FP16 mode without duplicating weights in memory.

Strength
1) An elegant nested encoding lets one FP16 checkpoint serve both FP8 and FP16 without extra scales/metadata, avoiding memory bloat common in co-deploying separate FP8/FP16 models. Exact FP16 recovery is guaranteed.
2) CUTLASS kernel pipelines reconstruction to hide latency; authors show cache-friendly FP8 layout (no scattered loads) and integrate everything into vLLM, enabling end-to-end serving experiments.
3) The paper goes beyond GEMMs: SLO-aware batching/precision scheduling reduces TTFT and improves SLO attainment under Azure traces, including multi-GPU settings.
4) Results span multiple model families/sizes (incl. 70B) and compare FP8 error against a strong baseline (DeepSeek-FP8)

Weakness
1) The method relies on most weights having |w|<=1.75 to fit E4M3; layers violating this become “exceptions” that fall back to FP16. The paper shows only a handful of such layers, but scenarios with heavier tails (other checkpoints/training recipes) could reduce FP8 coverage.
2) While modest (~3.9-4.5%), it may still matter for latency-sensitive paths; the kernel optimizations are hardware-specific and could shift with future architectures.
3) FP8 relative error is “close but slightly higher” than DeepSeek-FP8 in some layers; the paper argues it’s negligible at model level, but a deeper analysis of pathological cases (reasoning/coding long-context) would be valuable.

Recommendation:
Unifying FP8/FP16 from a single checkpoint, while preserving exact FP16—solves a real deployment problem. The combination of kernel work, vLLM integration, and SLO-aware scheduling demonstrates tangible end-to-end benefits (TTFT/TPOT & throughput) beyond microbenchmarks. I stop short of spotlight: while impactful, parts of the empirical story (BF16 conversion path, edge cases with heavy-tailed weights, deeper long-context/coding analyses) could be more comprehensive.

Net assessment after rebuttal:  Most substantive concerns (applicability to larger models, serving metrics beyond GEMM, BF16 path clarity, comparisons to FP8 baselines, per-layer error) were addressed with additional results and discussion; only the “why keep FP16 if FP8 can suffice” question remained partially debated. Given the single-checkpoint deployment win, strong system integration, and measured end-to-end benefits, I recommend Accept (Poster).
Decision: Accept (poster)