FP-Quant support
Author: BlackSamorez
Pull request: yes

# This PR adds support for the FP-Quant method.

The goal of this PR is to integrate inference and training support for the FP-Quant method that utilizes the Hadamard Transform for efficient weights+activations quantization. When using it with MXFP4 and MSE-based scaling, it implements [Quartet forward pass](https://arxiv.org/abs/2505.14669). We're also working on adding NVFP4 support and backward pass support.

Currently, we're working on the kernels [here](https://github.com/IST-DASLab/qutlass), and the integration [here](https://github.com/IST-DASLab/FP-Quant).

Installation:
 1. Install `qutlass`: `git clone https://github.com/IST-DASLab/qutlass.git && cd qutlass && pip install --no-build-isolation .`
 2. Install `fp_quant`: `pip install fp_quant`

Usage:
 1. Use as JIT quantization from any BF16 model by passing `quantization_config=FPQuantConfig()`
 2. Calibrate with GPTQ with [the repo](https://github.com/IST-DASLab/FP-Quant) with `--real_quant`.
 3. Use pre-quantized models from hub: coming soon...

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @mekkcyber

Hi @BlackSamorez, I'm really looking forward to experimenting with this.

When can we expect to have the kernels public so we can begin testing, even if they are still WIP?

@MekkCyber Hi, thanks for reviewing this!
It took us a while, but all the kernels necessary for inference have been published: I've updated the PR description.
May I ask you to do another pass? Your previous comments mostly don't apply anymore because of refactoring. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38696). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc added docs, improved docstring, cleaned the code where you asked.


Actually, give me a minute. I'm adding Triton pseudo-quantization kernels for people without Blackwell GPUs to be able to evaluate the models (although without speedups).

Added pseudoquantization, updated requirements to run the method with it (doesn't require `qutlass` like that). Added pseudoquant tests. Updated the documentation.

@SunMarc please take a look at the new raised errors and and warning in the `quantizer_fp_quant.py`.

Should be good

One last nit, the build PR documentation is not passing:

```
    raise RuntimeError(
RuntimeError: The following files are not present in the table of contents:
- quantization/fp_quant
Add them to ../transformers/docs/source/en/_toctree.yml.
```

Added it to toctree

@SunMarc it hit job cancellation somehow. Might need a restart. It should be good.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: fp_quant_integration

Merged ! Thanks for your work 

Hey @BlackSamorez, is there a way to make fp_quant compatible with py3.9 ? Our CI runs on this version but fp_quant requires 3.11 

I guess I'll have to remove match-case constructions and it'll work.
Why run on 3.9 in 2025 though?

We want to make sure that the min version of python that is maintained runs transformers correctly. When it will reach EOL, we switch to the next version