Step-5-Preview-NVFP4-Local

A 4-bit build of StepFun's Step 5 Preview sized for four 96 GB GPUs: 347.8 GB, against 394.2 GB for stepfun-ai/Step-5-Preview-NVFP4 (revision bc246cc2848e17b37f377eda8fd825c5b3018d1f), from which it is made.

• Routed experts are NVFP4 in all 88 MoE layers: layers 3 to 87 as StepFun quantized them, and layers 88 to 90, which StepFun shipped in BF16, quantized here the same way (NVFP4 weights, BF16 activations).
• The E4M3 block scales of every routed expert are stored in b12x's compressed scale format (CSF), the format introduced with nvidia/Qwen3.8-Flash-Next-FP4-Local, about half the size of the raw scales.
• Everything else (attention, shared experts, routers, dense layers, MTP layers, embeddings, lm_head, vision tower, tokenizer, chat template) is StepFun's NVFP4 release unchanged, in BF16 as StepFun shipped it.

Step 5 Preview is StepFun AI's model: a sparse mixture of experts with 600B total and 27B active parameters, text and image input, up to 1M tokens of context. Model card: stepfun-ai/Step-5-Preview. This repository is not affiliated with or endorsed by StepFun AI. Use is subject to the StepFun Model License 1.0 (LICENSE.txt, with StepFun's Chinese text in LICENSE-CN.txt).

Status: weights complete. KL divergence against the BF16 model has not been measured: StepFun's BF16 reference implementation, run layer by layer here, did not pass a causality check (see Fidelity), so no fidelity figure is published. The serving image in serving/ has been checked on CPU only and has not been run on a GPU. No accuracy or serving claim is made here.

Modifications

Notice of changes under section 2(a) of the StepFun Model License. Made by Brandon M. Music on 2026-10-16 to stepfun-ai/Step-5-Preview-NVFP4 at revision bc246cc2848e17b37f377eda8fd825c5b3018d1f:

1. Layers 3 to 87: each model.layers.N.moe.{gate,up,down}_proj.weight_scale (F8_E4M3 [352, R, C]) is stored as three CSF tensors that encode the same scale bytes (see Format).
2. Layers 88 to 90: each BF16 model.layers.N.moe.{gate,up,down}_proj.weight ([352, R, K]) is replaced by NVFP4: weight (U8 [352, R, K/2], E2M1 pairs), weight_scale_2 (F32 [352]) and CSF tensors holding the E4M3 block scales. These layers have no input_scale; their activations stay BF16.
3. config.json and hf_quant_config.json: quantization config MIXED_PRECISION with one quantized_layers entry per MoE layer, model.layers.N.moe.experts: {"quant_algo": "NVFP4", "group_size": 16, "weight_scale_encoding": "csf"} for N = 3 to 87 and {"quant_algo": "W4A16_NVFP4", ...} for N = 88 to 90. The three *layers.N.moe.experts* entries for 88 to 90 are removed from StepFun's ignore / exclude_modules lists; the rest of those lists is unchanged.
4. model.safetensors.index.json and SHA256SUMS are regenerated; csf-report.json, bf16-experts-nvfp4-report.json and bf16-layer-sensitivity.json (integrity and weight-level reports) and serving/ are added; this README replaces StepFun's card.

LICENSE.txt and LICENSE-CN.txt are StepFun's, unchanged.

Size

	stepfun-ai/Step-5-Preview-NVFP4	This repository
Tensor bytes	394,164,831,232 (394.2 GB)	347,771,251,120 (347.8 GB)
Layers 3-87 routed-expert block scales	35,295,068,160	18,154,837,272 (CSF, 51.44%)
Layers 88-90 routed experts	39,862,665,216 (BF16)	10,609,303,320 (NVFP4 9,965,666,304 + CSF scales 643,637,016)
Saved		46,393,580,112 (46.4 GB, 11.8%)

At tensor parallel 4 the weights come to about 86.9 GB (81.0 GiB) per GPU. The b12x loader keeps the compressed scale planes resident and expands one MoE layer's scales into shared scratch when that layer runs, so the CSF saving applies to GPU memory as well as disk.

Layers 88-90

StepFun kept exactly these three layers' routed experts out of quantization in both of its quantized releases: the FP8 checkpoint (stepfun-ai/Step-5-Preview, whose modules_to_not_convert starts with them) and the NVFP4 one. In the architecture they are ordinary routed-expert layers (same shapes, router, sliding-window attention and SwiGLU limit as their neighbours); they are the last three MoE layers before the final dense layer and the output head. StepFun does not say why. Here they are NVFP4 to fit four 96 GB GPUs.

Their weights are not harder to quantize than the layers StepFun did quantize: the same NVFP4 rounding gives a weight NMSE of 0.903% on layers 88-90 (all 352 experts, against BF16) and 0.900% on layers 85-87 (32 experts per layer, against StepFun's FP8 copy), with the same kurtosis and tails (bf16-layer-sensitivity.json, bf16-experts-nvfp4-report.json). That is a weight-level comparison only; their effect on the output would be part of the KL-divergence measurement, which has not been made.

Format

For each routed-expert scale tensor <p>.weight_scale with shape [E, R, C] (E = 352 experts; gate and up R = 1536, C = 256; down R = 4096, C = 96; one E4M3 byte per 16 weights):

Tensor	Dtype, shape	Contents
<p>.weight_scale.nvfp4_csf_fixed	U8 [E, R/16, 16 * (1 + C/2)]	per 16-row slab: 16 row bases, then each row's 4-bit codes, two per byte, low nibble first
<p>.weight_scale.nvfp4_csf_exceptions	U32 [n]	(row * C + col) | byte << 24, grouped by expert, row-major within an expert
<p>.weight_scale.nvfp4_csf_starts	I64 [E + 1]	expert e owns exceptions[starts[e]:starts[e + 1]]

A scale byte is its row's base plus its 4-bit code, unless an exception at that position replaces it with the stored byte. Each row's base is the lowest value whose 16-value window covers the most of the row's bytes (bases are 0 to 240). The three tensors are exactly the components of b12x's CsfSource((fixed, exceptions), starts, R, C, "nvfp4", E, "csf"); a single expert is fixed[e] with exceptions[starts[e]:starts[e + 1]], the per-matrix form that nvidia/Qwen3.8-Flash-Next-FP4-Local stores. Step-5's fused [352, ...] expert layout is kept for every tensor. NVFP4 weights dequantize as E2M1 value x E4M3 block scale x weight_scale_2.

Fidelity

The fidelity measure for this checkpoint is KL divergence against the BF16 model (stepfun-ai/Step-5-Preview-BF16, revision 7734b7ffac118d70d8398bfa3eb0fab007eb45b1). It has not been measured. StepFun's card states that its full BF16 workflow has not been validated end to end, and here its BF16 reference implementation, run layer by layer, failed a causality check: in its sparse-attention layers the outputs at positions 0 to 63 changed when only tokens 64 to 127 changed. Its teacher predictions on agentic and coding text were also not plausible. A KL divergence computed against that teacher would not mean anything, so none is reported; the measurement waits for a BF16 reference that passes.

Conversion integrity checks (these confirm the files were written as intended; they are not a fidelity measurement): the source files match StepFun's SHA256SUMS (102 of 102); every CSF tensor passes b12x CsfSource.validate() and decodes, with an independent decoder, to the scale bytes it encodes, checked before and after writing; every packed NVFP4 matrix of layers 88-90 decodes exactly to the quantizer's output (3,168 of 3,168); all other tensors are byte-identical to StepFun's.

Serving

This checkpoint needs a vLLM build with both:

• Step-5 support: upstream pull request vllm-project/vllm#61727 (StepFun's vllm/vllm-openai:step-5-preview image is built from it), and
• b12x's NVFP4-CSF expert loader (weight_scale_encoding: "csf", quantization modelopt_mixed, MoE backend b12x), as in Local Inference Lab's vLLM images.

serving/ holds an image that combines both on Local Inference Lab's vLLM (serving/step5-lil-tp4-dev.tar, see serving/SERVING.md). It has been checked on CPU only. Upstream vLLM and the step-5-preview image cannot read the CSF scale tensors; use stepfun-ai/Step-5-Preview-NVFP4 with those.

Credits

• Step 5 Preview and its NVFP4 checkpoint: StepFun AI. The NVFP4 export was produced with NVIDIA Model Optimizer 0.47.0, per the checkpoint's hf_quant_config.json.
• CSF scale format, encoder, validator and loader: b12x, in Local Inference Lab's flashinfer (flashinfer/experimental/b12x), the format first shipped with nvidia/Qwen3.8-Flash-Next-FP4-Local.