Add FP8 kernel acceleration for compressed-tensors quantized models
Author: jiqing-feng
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=45699)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45699)
<!-- ci-dashboard-badge:end -->

## What does this PR do?

This PR adds native FP8 matmul kernel support for compressed-tensors FP8 quantized models in transformers. Previously, compressed-tensors FP8 models were loaded via the `compressed-tensors` library and dequantized back to FP16/BF16 for inference. With this change, FP8 weights are kept in FP8 format and inference uses hardware-accelerated FP8 matmul kernels (`torch._scaled_mm` on XPU, `fbgemm.f8f8bf16_rowwise` on CUDA).

### Key changes:

**New file: `src/transformers/integrations/compressed_tensors_fp8.py`**
- `CTFP8Linear`: FP8 linear layer that stores weights in FP8 and uses row-wise FP8 matmul kernels. Activations are dynamically quantized per-row via `quantize_fp8_per_row`.
- Weight converters (`CompressedTensorsScaleConvert`, `CompressedTensorsFp8Dequantize`) to handle the checkpoint format conversion (e.g. `weight_scale` → `weight_scale_inv`).
- `CTFP8PerRowQuantize`: Online quantization support — quantize BF16 weights to FP8 per-row on-the-fly during model loading.

**Modified: `src/transformers/quantizers/quantizer_compressed_tensors.py`**
- `CompressedTensorsHfQuantizer` now detects FP8 quantization configs (`float` type, `num_bits=8`) and automatically routes to the FP8 kernel path when GPU/XPU is available. Falls back to the default compressed-tensors dequantize path on CPU.
- Added `get_weight_conversions()` and `get_quantize_ops()` to support both pre-quantized loading and online quantization.
- No changes to the non-FP8 code path — existing INT8/INT4 compressed-tensors models are unaffected.

**Modified: `src/transformers/quantizers/auto.py`**
- Minor formatting change (no functional change).

## Supported models

- **Per-channel dynamic**: e.g. [RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic](https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic)
- **Per-tensor static**: e.g. [RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8](https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8)
- **Online quantization**: Any BF16 model can be quantized to FP8 on-the-fly by passing a `CompressedTensorsConfig` with FP8 quantization scheme.

## Usage

### Pre-quantized model (no config needed)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
```

### Online quantization
```python
from transformers import AutoModelForCausalLM, CompressedTensorsConfig
from compressed_tensors.quantization import QuantizationScheme, QuantizationArgs, QuantizationType, QuantizationStrategy

ct_config = CompressedTensorsConfig(
    config_groups={
        "group_0": QuantizationScheme(
            weights=QuantizationArgs(
                num_bits=8, type=QuantizationType.FLOAT, strategy=QuantizationStrategy.CHANNEL,
            ),
        ),
    },
    run_compressed=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    quantization_config=ct_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
```

## Devices
- **XPU** (Intel Data Center Max / Arc): uses `torch._scaled_mm`
- **CUDA** (SM89+): uses `fbgemm.f8f8bf16_rowwise`
- **CPU**: falls back to default compressed-tensors dequantize path


@sywangyi

Comments:
cc @SunMarc 

Hi @SunMarc . Please check it the integration is ok. I'll clean the tests and doc after you approved the integration.

Hi @SunMarc . Would you please review the PR? Thanks!

Hi @Rocketknight1 . It seems that @SunMarc  does not have bandwidth to review this PR. Would you please help to review the PR? Thanks!

Hi @IlyasMoutawwakil, thanks for the review! I've addressed most of the feedback. Here's an update:

## Bug fix: `scale_b` must be contiguous for `torch._scaled_mm`

I found a bug introduced when removing the `.contiguous()` call on `scale_b` per your suggestion. For **per-tensor static** models (e.g. `RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8`), the scale shape is `(1, 1)` and needs to be expanded to `(1, out_features)`. `expand()` creates a stride-0 view, but `torch._scaled_mm` requires contiguous scale tensors:

```
RuntimeError: Invalid scaling configuration.
- For RowWise scaling, scale_b should be contiguous.
Got scale_b.stride()=[1, 0]
```

I've added `.contiguous()` back **only for `scale_b`** (not for the input). The scale tensor is tiny (~16 KB for a 4096-dim layer), so the cost is negligible. The input `.contiguous()` was correctly replaced with `.reshape()` as you suggested.

## Changes addressing review comments

1. **Class naming**: Renamed `CTFP8Linear` → `CompressedTensorsFP8Linear` to avoid confusion with CTransformers
2. **Kernel caching**: Switched from `get_kernel()` to `lazy_load_kernel()` for cached kernel loading
3. **`quantize_fp8_per_row` reference**: Removed from `__init__` parameter, now called via `_get_quantize_fp8_per_row()` at forward time
4. **Input `.contiguous()`**: Replaced with `.reshape()` which handles both contiguous and non-contiguous inputs correctly
5. **Docstring**: Fixed the description — this PR enables FP8 inference (not just dequantization)
6. **`_original_config_dict` in auto.py**: Removed

Hi @SunMarc @IlyasMoutawwakil . Please let me know if it's okay to merge this PR. Thanks! I can add tests and clean docs in the next PR if it is fine for you.

Hi @SunMarc . Is it okay to merge the PR?

Hi @Rocketknight1 @ArthurZucker . Do you have bandwidth to review this PR? Thanks!

CI failures are pre-existing on main (training_ci, torch, tensor_parallel_ci all fail on main nightly too), unrelated to this PR's changes.

Yep having al ook 

@jiqing-feng , pls update according to steve's comment, thx

Hi @stevhliu . I have fixed all your comments. Please review the new changes. Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45699). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @ArthurZucker . Would you please review my new commit to check if it fixed your comments? Thanks!

Hi @SunMarc . Would you please let me know what need to be changed before merging? Thanks!

Hi @SunMarc, thanks for the review! Addressed everything; kept the MoE dequant-before-merge on purpose (explained below).

**Done**
- **No more online quant** — removed `CompressedTensorsFP8PerRowQuantize`; `get_quantize_ops` returns `None` and `param_needs_quantization` is `False` for FP8. Online FP8 → use finegrained-fp8.
- **`dequantize` arg** — added `dequantize: bool = False` to `CompressedTensorsConfig` (like the other methods), wired through to `replace_with_compressed_tensors_fp8_linear`. `run_compressed` no longer drives this.
- **(1)** dropped the redundant `.to(float32)` (`weight_scale` is already f32).
- **(2)** added an `is_per_tensor` variable.
- **(3)** input is reshaped to 2D once, shared by both paths.
- **(4)** removed `with torch.device("meta")`.
- **(5)** moved `_is_fp8_config` → `CompressedTensorsConfig.is_fp8`.
- **(6) / (7)** `is_trainable` / `is_qat_trainable` now return `dequantize` for FP8 (trainable only on the dequantized BF16 path).
- **(8)** kept `input_scale` — the converter was dead code, removed it.
- **(9)** kept `weight_scale` (no more `weight_scale_inv`); the converter only reshapes.

Side effect: `CompressedTensorsFp8Dequantize.reverse_op` is now `_IdentityOp()` (we never re-quantize on save).

**The MoE dequant (comments 10 & 11)**

MoE checkpoints store per-expert weights+scales, but transformers merges the experts (`stack`/`cat`) into a single **3D packed `nn.Parameter`** — not an `nn.Linear`, so it can't hold a `weight_scale`, and the per-expert scales differ so they can't survive the merge. The only correct option is to **dequantize each expert (`weight * weight_scale`) to BF16 before the merge**: `update_weight_conversions` prepends `CompressedTensorsFp8Dequantize` to the merging converters, pairs weight+scale by expert index, drops the scales. Without it, MoE FP8 loading crashes. (Attention/router linears in the same model still stay FP8.) The `dequantize=True` flag reuses this same op to also fold plain linears to BF16 — that's the answer to comment 10.

**Known limitation:** since merged experts land in BF16, this PR **doesn't save memory on MoE expert weights** (the bulk of params) — only dense models and the attention/router part of MoE benefit. Not fundamental: experts *could* stay FP8 with (1) a 3D FP8 expert param, (2) a stacked per-expert scale tensor, (3) a grouped scaled-mm in the MoE forward. That's a bigger change, so I'd suggest a follow-up and keeping this dequant path as the correct baseline.

Happy to adjust naming or split differently — let me know!


Hi @SunMarc . I've fixed all your comments. Please check it. Thanks!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: compressed_tensors_integration

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45699)
**Latest run:** [28875849313:2](https://github.com/huggingface/transformers/actions/runs/28875849313)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 172,665 | **Failures:** 1 | **Duration:** 21h 45m

<!-- ci-dashboard-recap:end -->

@SunMarc, may I know your plan on this? Go with this PR or your PR https://github.com/huggingface/transformers/pull/47216? thx very much.