FineGrainedFP8 DeepGEMM path returns silently wrong results on SM100 (UE8M0 scale rounding without requantization)
Author: keunhong
Pull request: no

## System info

- transformers 5.12.1, torch 2.10.0+cu130, kernels 0.12.3
- Hub kernel: `kernels-community/deep-gemm`, pinned version 2 (rev `9590415046fa`, internal `__version__` 2.5.0)
- GPU: NVIDIA B200 (SM100), CUDA 13.0
- Model: `Qwen/Qwen3-4B-FP8` (fine-grained FP8, dynamic activation scheme, fp32 `weight_scale_inv`, `weight_block_size=[128,128]`)

## Bug

On SM100, `fp8_linear` prefers the DeepGEMM hub kernel (`integrations/finegrained_fp8.py`, `deepgemm_preferred`). DeepGEMM's only SM100 FP8 GEMM (`sm100_fp8_gemm_1d1d`) consumes UE8M0 (power-of-two) scale factors. transformers satisfies that contract by ceil-rounding the fp32 dequant scales to the next power of two (`integrations/deepgemm.py::_ceil_to_ue8m0` via `_coerce_sf_for_kernel`) — but it never requantizes the data those scales describe:

- checkpoint weights were quantized against the original fp32 `weight_scale_inv`;
- activations are quantized on the fly with the raw fp32 scale (`_select_fp8_cast_kwargs` returns `use_ue8m0=False` for fp32-scale checkpoints).

Rounding a dequant scale up by a factor in [1, 2) without requantizing the payload inflates every 128-block partial sum by that factor, independently per activation row-block and per 128x128 weight block. The result is finite and warning-free, and completely wrong.

Measured on `Qwen/Qwen3-4B-FP8`:

- single `FP8Linear` forward: relative error ~0.8-1.1 vs the fp32-scale reference, output gain ~1.8-2.1x;
- end-to-end hidden states (layer 35 residual): per-token cosine vs correct ~0 (mean -0.16 to -0.03), norms inflated 7-15x.

On SM90 the same dispatch selects `sm90_fp8_gemm_1d2d`, which consumes fp32 scales exactly, so H100 results are correct — the corruption is Blackwell-only, which makes it easy to miss in CI.

`TRANSFORMERS_DISABLE_DEEPGEMM_LINEAR=1` (Triton fallback) produces correct results on both architectures and is the workaround we deployed.

## Reproduction

```python
import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Qwen/Qwen3-4B-FP8"
PROMPT = "A red canoe rests on the shore of a mountain lake at dawn."

def hidden(disable_deepgemm: bool) -> torch.Tensor:
    os.environ["TRANSFORMERS_DISABLE_DEEPGEMM_LINEAR"] = "1" if disable_deepgemm else "0"
    tok = AutoTokenizer.from_pretrained(MODEL)
    model = AutoModelForCausalLM.from_pretrained(MODEL, device_map="cuda")
    ids = tok(PROMPT, return_tensors="pt").to("cuda")
    with torch.inference_mode():
        out = model.model(**ids, output_hidden_states=True, use_cache=False)
    h = out.hidden_states[-1].float().cpu()
    del model
    torch.cuda.empty_cache()
    return h

# Run on an SM100 GPU:
a = hidden(disable_deepgemm=True)   # Triton, fp32 scales, correct
b = hidden(disable_deepgemm=False)  # DeepGEMM sm100_fp8_gemm_1d1d, UE8M0-rounded scales
cos = torch.nn.functional.cosine_similarity(a.flatten(), b.flatten(), dim=0)
print("cosine:", cos.item(), "norm ratio:", (b.norm() / a.norm()).item())
# Observed: cosine ~0, norm ratio ~7-15. Expected: ~1.0, ~1.0.
```

## Expected behavior

Either requantize weights and activations against the UE8M0 scales before calling the SM100 kernel (vLLM does this on load — see vllm-project/vllm#37804 for the accuracy discussion even *with* proper requantization), or fall back to the Triton kernel on SM100 until then. A silent >100% error should not be reachable from `from_pretrained` defaults.



Comments:
Prior-art notes after more digging:

- #46209 (Qwen3-0.6B-FP8 on B200, incoherent output, open since May) looks like this same defect observed through generation output.
- #46818 (merged, unreleased) already introduces `_assert_sm100_scales_are_ue8m0`, whose docstring describes exactly this silent-rounding corruption -- but it guards only the two MoE experts forwards. `deepgemm_fp8_fp4_linear`, which every dense `FP8Linear` hits, still calls `_coerce_sf_for_kernel` unguarded on current main.
- Ecosystem precedent for the fix direction: vLLM shipped this same bug class (Qwen3-30B-A3B-FP8 scoring GSM8K 0.0 on B200) and fixed it with UE8M0 weight requantization at load (vllm-project/vllm#22399); they later measured 5-12pp degradation for Qwen3.5 FP8 even with proper requantization (vllm-project/vllm#37804) and ended up auto-disabling DeepGEMM for those model types on Blackwell (vllm-project/vllm#38083). sglang followed the same arc (sgl-project/sglang#13601). DeepSeek's upstream DeepGEMM now device-asserts on non-power-of-two scales (deepseek-ai/DeepGEMM#364), but the hub build removed the python-side assert for cudagraph compatibility (huggingface/kernels-community#940), so nothing fires.

Given vLLM's accuracy data, extending the #46818 guard to the dense path (fail loud) or de-preferring DeepGEMM in `deepgemm_preferred` for fp32-scale checkpoints on SM100 (falling through to the Triton kernel, which handles fp32 scales exactly) both look preferable to adding requantization for these checkpoints.