[BUG] FP8 model cannot inference on B200 (SM>=100)
Author: xin3he
Pull request: no
Labels: name: bug

### System Info

transformers==5.9.0
kernels==0.14.1

### Who can help?

@SunMarc @MekkCyber

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I cannot get correct output with kernels==0.12.0, and got error using kernels==0.14.1.
I'm not sure who is responsible for the kernel package. it's the default backend in transformers.

https://huggingface.co/Qwen/Qwen3-0.6B-FP8

```
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-0.6B-FP8"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=8
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

# parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

print("thinking content:", thinking_content)
print("content:", content)
```

### Expected behavior

should have a reasonable output.

Comments:
Hi @SunMarc @MekkCyber,

Wanted to mention a potentially related issue:
intel/auto-round#1851 ("CUDA CI Failed on SM120")

It shows several similar failures on newer Blackwell architectures (SM120), including:

* `kernels` FP8 backend import failures
* `flashinfer` / CUTLASS DSL errors on newer CUDA requirements
* incorrect generation outputs on quantized inference paths
* failures across multiple inference stacks (`sglang`, `vllm`, AutoRound)

This seems consistent with the behavior reported here on B200/SM100:

* `kernels==0.12.0` → incorrect output
* `kernels==0.14.1` → runtime failure

So the issue may be broader SM>=100 compatibility across some underlying CUDA/Triton/kernel backends rather than transformers logic alone.

Given that, would a transformers-side PR adding:

* explicit SM capability detection,
* clearer unsupported-hardware errors/warnings,
* or graceful fallback behavior for FP8 paths

be useful as an interim improvement while backend support matures upstream?

I’d be happy to work on the detection/guard path in `quantizer_fp8.py` if that direction makes sense.


Can you share your traceback for both versions of kernels @xin3he ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@SunMarc 
sorry I missed the notificaion.

- with latest kernels==0.16.0

<details>

```
Traceback (most recent call last):
  File "/workspace/test_bug.py", line 7, in <module>
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 359, in from_pretrained
    explicit_local_code = has_local_code and not _get_model_class(
                                                 ^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 179, in _get_model_class
    supported_models = model_mapping[type(config)]
                       ~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 606, in __getitem__
    return self._load_attr_from_module(model_type, model_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 620, in _load_attr_from_module
    return getattribute_from_module(self._modules[module_name], attr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 532, in getattribute_from_module
    if hasattr(module, attr):
       ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2262, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2496, in _get_module
    raise e
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2494, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 27, in <module>
    from ...activations import ACT2FN
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/activations.py", line 22, in <module>
    from .integrations.hub_kernels import use_kernel_forward_from_hub
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/integrations/hub_kernels.py", line 105, in <module>
    "cuda": LayerRepository(
            ^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/kernels/layer/layer.py", line 77, in __init__
    raise ValueError("Either a revision or a version must be specified.")
ValueError: Either a revision or a version must be specified.
```

</details>

- kernels==0.14.0

<details>

```
Loading weights: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 507/507 [00:00<00:00, 3463.94it/s]
Downloading (incomplete total...): 100%|????????HTTP Error 429 thrown while requesting GET https://huggingface.co/api/resolve-cache/kernels/kernels-community/deep-gemm/326730c8a64c45acf7fb11f7320fac6ca4c70031/build%2Ftorch211-cu130-x86_64-linux%2Finclude%2Fthird-party%2Fcutlass%2Finclude%2Fcutlass%2Farch%2Fmemory_sm80.h
Rate limited. Waiting 124.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting GET https://huggingface.co/api/resolve-cache/kernels/kernels-community/deep-gemm/326730c8a64c45acf7fb11f7320fac6ca4c70031/build%2Ftorch211-cu130-x86_64-linux%2Finclude%2Fthird-party%2Fcutlass%2Finclude%2Fcutlass%2Farch%2Fmma_sm100.h
Rate limited. Waiting 123.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting GET https://huggingface.co/api/resolve-cache/kernels/kernels-community/deep-gemm/326730c8a64c45acf7fb11f7320fac6ca4c70031/build%2Ftorch211-cu130-x86_64-linux%2Finclude%2Fthird-party%2Fcutlass%2Finclude%2Fcutlass%2Farch%2Fmma.h
Rate limited. Waiting 123.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/326730c8a64c45acf7fb11f7320fac6ca4c70031/build/torch211-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/arch/mma_sm50.h
Rate limited. Waiting 123.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/326730c8a64c45acf7fb11f7320fac6ca4c70031/build/torch211-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/arch/mma_sm70.h
Rate limited. Waiting 123.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting GET https://huggingface.co/api/resolve-cache/kernels/kernels-community/deep-gemm/326730c8a64c45acf7fb11f7320fac6ca4c70031/build%2Ftorch211-cu130-x86_64-linux%2Finclude%2Fthird-party%2Fcutlass%2Finclude%2Fcutlass%2Farch%2Fmma_sm60.h
Rate limited. Waiting 123.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting GET https://huggingface.co/api/resolve-cache/kernels/kernels-community/deep-gemm/326730c8a64c45acf7fb11f7320fac6ca4c70031/build%2Ftorch211-cu130-x86_64-linux%2Finclude%2Fthird-party%2Fcutlass%2Finclude%2Fcutlass%2Farch%2Fmma_sm61.h
Rate limited. Waiting 123.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/326730c8a64c45acf7fb11f7320fac6ca4c70031/build/torch211-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/arch/mma_sm75.h
Rate limited. Waiting 123.0s before retry [Retry 1/5].
Downloading (incomplete total...): 100%|????????????????????????????????????????????????????????????????????????????????????????????| 18.2M/18.2M [00:28<00:00, 4.14MB/s]
Fetching ... files: 374it [00:24, 22.96it/s]

```
</details>

- kernels==0.12.0
<details>
```
Loading weights: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 507/507 [00:00<00:00, 2733.92it/s]
Fetching ... files: 1446it [01:02, 22.97it/s]????????????????????????????????????????????????????????????????????????????????????| 37.6M/37.6M [01:07<00:00, 219kB/s]
Download complete: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 37.6M/37.6M [01:18<00:00, 219kB/s]thinking content: 
content: ISM�?.igr�?ized box 
Download complete: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 37.6M/37.6M [01:31<00:00, 409kB/s]
```
</details>



We hit what looks like this same defect and root-caused it in #47030: on SM100, the finegrained-fp8 path prefers the `kernels-community/deep-gemm` hub kernel, whose only SM100 FP8 GEMM consumes UE8M0 (power-of-two) scale factors. `_coerce_sf_for_kernel` satisfies that by ceil-rounding the checkpoint's fp32 dequant scales to the next power of two without requantizing the weights or activations, so every 128-block partial sum is inflated by a random factor in [1, 2) -- finite, warning-free, and wrong. SM90 dispatches to the fp32-scale `sm90_fp8_gemm_1d2d` kernel and is correct, which matches this reproducing only on B200.

Two things that may help confirm on your side: `TRANSFORMERS_DISABLE_DEEPGEMM_LINEAR=1` (available in v5.11+) routes to the Triton finegrained-fp8 kernel, which handles fp32 scales exactly -- output should become coherent; and #47030 has a self-contained repro comparing the two paths on one prompt.

Note that #46818 (merged, not yet released) introduces `_assert_sm100_scales_are_ue8m0` whose docstring names exactly this silent-rounding corruption, but wires it only into the two MoE experts forwards -- the dense `deepgemm_fp8_fp4_linear` path that `FP8Linear` uses is still unguarded on main.