LibertAIDAI

/

GLM-5.3-Flash-NVFP4

like 52

Follow

LibertAI 45

Thank you for reporting this — it is our bug, and your report plus an independent reproduction on our own sm_120 deployment let us find it. Fixed on main a few minutes ago; please re-download model-input-scales.safetensors.

What was wrong. Two days ago this checkpoint carried no input_scale at all, which makes vLLM's ModelOptNvFp4FusedMoE fold uninitialised memory into the dequant alphas and zero every expert. We "fixed" that by shipping input_scale = 1.0 placeholders, arguing that the scale cancels analytically so any value in a wide band is correct. That argument was wrong, and it produced exactly the symptom you describe.

There are two bounds, and they pull in opposite directions:

no clipping:            input_scale >= amax_act   / 2688
no block-scale underflow: input_scale <= amax_block / 0.1     <-- per 16-element BLOCK

We checked the second against the tensor amax (~1.7) and concluded 1.0 was safe. It is per block. At input_scale = 1.0, every activation block whose amax is below 0.1 has its fp8 block scale flushed to zero. Which blocks those are depends entirely on the input — so the model is fine most of the time and degenerates on some inputs, which is precisely "repeats itself time to time". It gets worse as context grows, because more tokens means more low-magnitude blocks.

For scale: the real calibrated values for this model span 5.58e-04 to 3.72e-02 with a median of 1.58e-03. Our placeholder was 632x the median.

Fixes, in order of preference:

1. Re-download model-input-scales.safetensors from main. It now carries real per-projection calibrated values (36,288 of them). The underflow threshold moves from amax_block >= 0.1 to >= 1.6e-04.
2. --moe-backend marlin — dequantises the weights and never reads an activation scale, so it sidesteps the whole path. Costs a bf16 repack.

One question, because it changes the diagnosis completely: your --served-model-name says RedHatAI/GLM-5.3-Flash-NVFP4 but the path is /models/GLM-5.3-Flash-NVFP4. Which repo did you actually download? If you are on RedHat's checkpoint, it is W4A4 with its own calibrated activation scales and none of the above applies — in that case please say so and we will look again.

Also worth recording: this is the datacenter-Blackwell confirmation our card said we expected but had not seen. The trigger is the checkpoint, not the architecture — your GB300 and our sm_120 box hit the same thing.

Updated the model, will update here again if it happens.
Updated --served-model-name because I was testing something with their model as well, didn't want to change the code.
Actually, their model had the same issue as well.

Can confirm - fixed on the GB300