beyoru

/

KAT-Coder-V2.5-Dev-VL-Flash

like 0

Thanks for the report. This is an SGLang limitation, not a problem with the checkpoint.

Root cause. SGLang gates its whole activation-quantized branch on the top-level format field:

# sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py
quant_format = cast(str, config.get("format"))          # top-level only
...
if is_activation_quantization_format(self.quant_format):
    if self._is_fp4a4_nvfp4(...): return CompressedTensorsW4A4Fp4()
    if self._is_fp8_w8a8(...):    return CompressedTensorsW8A8Fp8(...)
raise NotImplementedError("No compressed-tensors compatible scheme was found.")

_ACTIVATION_QUANTIZATION_FORMATS (in compressed_tensors/utils.py) contains naive-quantized,
int-quantized, float-quantized, nvfp4-pack-quantized — but not mixed-precision.

This checkpoint has two config groups, so compressed-tensors sets the top-level format to
mixed-precision and puts the real format on each group:

top-level        : mixed-precision        -> is_activation_quantization_format = False  ← gate fails here
group_0 (FP8)    : float-quantized        -> True
group_1 (NVFP4)  : nvfp4-pack-quantized   -> True

SGLang never falls back to the per-group format, so it skips both branches and hits the raise.
mixed-precision is a valid member of CompressionFormat in the compressed-tensors library itself.

Verified by patching only that list: the error changes from
No compressed-tensors compatible scheme was found to a later, unrelated one, and
_is_fp4a4_nvfp4(weight_quant, input_quant) returns True — i.e. the scheme is recognised
correctly once the gate is passed.

This affects every multi-group compressed-tensors checkpoint, including the whole Unsloth
Dynamic NVFP4 line, which also ships format: "mixed-precision".

Workarounds

1. Use vLLM (confirmed working on Blackwell, sm121):

vllm serve beyoru/KAT-Coder-V2.5-Dev-VL-Flash \
    --max-model-len 32768 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code

--max-num-batched-tokens 8192 is required: this is a hybrid attention + Gated-DeltaNet model, so
with prefix caching on, vLLM forces mamba_cache_mode='align' and raises the attention block size to
2096, which exceeds the default max_num_batched_tokens of 2048 and aborts startup.

2. One-line patch if you want to stay on SGLang:

# sglang/srt/layers/quantization/compressed_tensors/utils.py
_ACTIVATION_QUANTIZATION_FORMATS = [
    CompressionFormat.naive_quantized.value,
    CompressionFormat.int_quantized.value,
    CompressionFormat.float_quantized.value,
    CompressionFormat.nvfp4_pack_quantized.value,
    CompressionFormat.mixed_precision.value,      # <-- add
]

Per-layer scheme selection already works after that, since _get_scheme_from_parts uses the
matched group's weights / input_activations rather than the top-level format.

Also note: NVFP4 needs Blackwell (sm100/sm120). Your log shows nranks 2, so if either GPU is
pre-Blackwell you will hit Current platform does not support w4a4 nvfp4 quantization after the
gate. On Hopper or older, use the bf16 build
beyoru/KAT-Coder-V2.5-Dev-VL instead.

...

I've got 3 GPUs total in that one machine and all are Blackwell. I appreciate the heads up, however! I'll try the patch and see what happens.

So I was trying to split it across my two 5090s and while the patch got it past that snag, it OOMed no matter what I did. My RTX Pro 6000 throws this error:

[2026-07-28 13:46:27] Found local HF snapshot for beyoru/KAT-Coder-V2.5-Dev-VL-Flash at /root/.cache/huggingface/hub/models--beyoru--KAT-Coder-V2.5-Dev-VL-Flash/snapshots/d26364cc084a750acddccc1e7e1195c2b2417668; skipping download.
Multi-thread loading shards:   0% Completed | 0/5 [00:00<?, ?it/s][transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
[2026-07-28 13:46:29] Using default HuggingFace chat template with detected content format: openai
[2026-07-28 13:46:29] Auto-detected template features: reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None), reasoning_parser=qwen3, tool_call_parser=qwen3_coder
[2026-07-28 13:46:32] Parameter lm_head.weight_scale not found in params_dict
[2026-07-28 13:46:32] Parameter model.layers.0.linear_attn.in_proj_qkvz.weight_scale not found in params_dict
[2026-07-28 13:46:32] Parameter model.layers.0.linear_attn.in_proj_qkvz.weight_scale not found in params_dict
[2026-07-28 13:46:32] Parameter model.layers.0.linear_attn.out_proj.weight_scale not found in params_dict
[2026-07-28 13:46:32] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 4623, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 431, in __init__
    self.init_model_worker()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 847, in init_model_worker
    self.init_tp_model_worker()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 771, in init_tp_model_worker
    self.tp_worker = TpModelWorker(**worker_kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 308, in __init__
    self._init_model_runner()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 424, in _init_model_runner
    self._model_runner = ModelRunner(
                         ^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 357, in __init__
    self.initialize()
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 567, in initialize
    self.load_model()
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 935, in load_model
    loaded = load_model_with_memory_saver(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner_components/load_model_utils.py", line 230, in load_model_with_memory_saver
    model = loader.load_model(
            ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/model_loader/loader.py", line 796, in load_model
    self.load_weights_and_postprocess(
  File "/sgl-workspace/sglang/python/sglang/srt/model_loader/loader.py", line 828, in load_weights_and_postprocess
    model.load_weights(weights)
  File "/sgl-workspace/sglang/python/sglang/srt/models/qwen3_5.py", line 2137, in load_weights
    param = params_dict[name_mapped]
            ~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'model.layers.0.mlp.experts.w2_input_global_scale'

I've fixed it on the checkpoint side, please git pull / re-download config.json.

It also explains the OOM on your two 5090s, so that wasn't a memory-sizing problem at all.

Root cause

SGLang matches the ignore list by substring, not exact match:

# sglang/srt/layers/quantization/compressed_tensors/utils.py
def check_equal_or_regex_match(layer_name, targets):
    for target in targets:
        if _is_equal_or_regex_match(layer_name, target, check_contains=True):   # <-- vLLM does NOT set this
            return True

With check_contains=True, a non-regex ignore entry matches whenever it is a substring of the layer name. vLLM calls the same helper without it — which is exactly why vLLM loads this model and SGLang does not.

llm-compressor emits parent module names into ignore, and this checkpoint had 10,270 of them:

The prefixes line up because SGLang builds the LM with HF-style names — qwen3_vl.py: prefix=add_prefix("model.language_model", prefix) — so the name handed to the matcher is model.language_model.layers.0.mlp.experts.0.gate_proj, and model.language_model.layers.0.mlp.experts.0 is a substring of it.

Result: should_ignore_layer → True → get_moe_scheme() returns None → get_quant_method() falls back to UnquantizedFusedMoEMethod, which registers only w13_weight / w2_weight in bf16.

That single fallback produces all three of your symptoms:

1. OOM on 2× 5090 — the MoE is allocated in bf16 instead of NVFP4 (roughly the full-precision model), and this happens in create_weights, before any weight is read. 64 GB was never going to be enough, hence "no matter what I did".
2. KeyError: model.layers.0.mlp.experts.w2_input_global_scale on the RTX Pro 6000 — 96 GB is enough to allocate, so it reaches load_weights; the first expert tensor in sorted order is input_global_scale, and there is no matching param.
3. Parameter ...linear_attn.*.weight_scale not found — the linear_attn entry swallowed the three FP8 projections. (lm_head.weight_scale is unrelated and harmless: SGLang's ParallelLMHead isn't a LinearBase, so it's never quantized.)

Quantified against the actual tensor index: 30,810 quantized modules were being ignored under SGLang's matcher, 0 under vLLM's.

What changed in the checkpoint

config.json only — no weight file was touched, no scheme, target, group or bit-width changed. The 10,270 parent-module entries were collapsed into two anchored regexes:

"ignore": [ ...the 280 real leaf entries..., "re:.*\\.mlp\\.experts\\.\\d+$", "re:.*\\.linear_attn$" ]

$ anchors the match to the container itself, so the semantics are identical — every one of the original 10,550 entries still resolves to "ignored" — but the entries are no longer substrings of their own children. Verified both ways: 0 quantized modules ignored under either matcher, 0 previously-ignored modules lost. Bonus: config.json went from 589 KB to 21 KB.

I also added packed_modules_mapping to quantization_config:

"packed_modules_mapping": {
  "qkv_proj":     ["q_proj", "k_proj", "v_proj"],
  "gate_up_proj": ["gate_proj", "up_proj"],
  "in_proj_qkvz": ["in_proj_qkv", "in_proj_z"],
  "in_proj_ba":   ["in_proj_b", "in_proj_a"]
}

Without it, once the layers stop being ignored, SGLang raises ValueError: Unable to find matching target for ...linear_attn.in_proj_qkvz. SGLang fuses in_proj_qkv + in_proj_z into in_proj_qkvz, and unlike vLLM it reads packed_modules_mapping only from quantization_config in config.json, never from the model class (which does define it, at qwen3_5.py).

The previous config.json is archived at legacy/config.json if you want to diff.

You still need the SGLang patch

The mixed-precision gate from my earlier comment is still required — that one can't be fixed from the checkpoint. One correction to that comment: the list is inside the function, not module-level:

# sglang/srt/layers/quantization/compressed_tensors/utils.py
def is_activation_quantization_format(format: str) -> bool:
    _ACTIVATION_QUANTIZATION_FORMATS = [
        CompressionFormat.naive_quantized.value,
        CompressionFormat.int_quantized.value,
        CompressionFormat.float_quantized.value,
        CompressionFormat.nvfp4_pack_quantized.value,
        CompressionFormat.pack_quantized.value,
        CompressionFormat.mixed_precision.value,      # <-- add
    ]
    return format in _ACTIVATION_QUANTIZATION_FORMATS

With both in place the load path should resolve as: experts → CompressedTensorsW4A4Nvfp4MoE, shared_expert → W4A4 FP4 linear, GDN/attention projections → W8A8 FP8, router / norms / vision tower → bf16.

I don't have a Blackwell card to verify end-to-end, so this is source-level analysis against the commit your traceback matches — please pull and let me know what you get.

Upstream

Three genuine SGLang issues worth filing, all independent of this model:

1. check_contains=True in the ignore path diverges from vLLM — any compressed-tensors checkpoint with a parent module name in ignore gets silently de-quantized, and llm-compressor emits those routinely. This is the serious one: it fails as OOM or KeyError, never as "your model is unquantized".
2. mixed-precision missing from _ACTIVATION_QUANTIZATION_FORMATS — affects every multi-group checkpoint, including the Unsloth Dynamic NVFP4 line.
3. packed_modules_mapping is read from config.json but never from the model class attribute.

Ok - I'll give it a go

So there is no flashqla prefill backend. I let it pick triton and it hung putting together the graph.. About 30-45m until I just killed it. It kind of just hangs on every prefill backend. Is there some fork this needs to run? I'm gonna give no-cuda-graphs a go and if that doesn't do it I'm gonna throw in the towel.

flashqla isn't in upstream SGLang -it's #31020, still open, and it's optional either way (a prefill speedup on the GDN layers, not a requirement). Also note it's --linear-attn-prefill-backend, a different flag from --attention-backend, the Triton one you switched. Neither is what's hanging you.

The hang is after backend init, which is why it looks the same on every backend. Two things and I can tell you which:

• last log line before it goes quiet Capture cuda graph begin or a torch.compile line?
• py-spy dump --pid <scheduler pid> while it's stuck

--disable-cuda-graph is a good next move. If it serves, come back with --cuda-graph-max-bs 4 instead of leaving graphs off.

One check: #32736 isn't merged. If you re-pulled SGLang after patching, both edits are gone and you're back to a bf16 MoE same OOM, no error.