Enable gptqmodel
Author: jiqing-feng
Pull request: yes

We are going to replace `auto_gptq` with `gptqmodel`. Start with the quantizer check, and also need to change the optimum: [https://github.com/huggingface/optimum/pull/2064](https://github.com/huggingface/optimum/pull/2064).

We intended to deprecate AutoGPTQ in this PR, but considering users' behavior, we would like to keep the support for auto_gptq for the next few versions and give a warning for deprecating.

Comments:
cc @SunMarc @MekkCyber 

@SunMarc GPTQModel is intended to replace AutoGPTQ entirely due to lack of progress in that repo for many reasons but for the sake of compat, they can co-exist in parallel until this integration is merged, everything is stable/tested, and maybe later we can add init a deprecation plan of AutoGPTQ which is no longer actively developed and/or maintained.

Hey @jiqing-feng, thanks for adding `gptqmodel` LGTM ! Could you update the PR description and title to make them clearer? Thanks 😊

@SunMarc Current PR in the current state is not passing our internal tests. @jiqing-feng Will merge some of our changes in that will pass both inference/quant tests. Please delay your review until then since there are substantial changes, relative to the code/PR currently.

Testing changes contain:

Refactor: CPU tests do not need @require_torch_gpu
GPTQ lib: @require_gptq means we can run these tests with gptqmodel or auto-gptq
Default model: we default run llama in tests instead of bloom because it's more common.

@SunMarc  Review can start at `optimum` first. I will write up a detailed explainer at the `optimum` pr on some obvious small/large change we pushed. 

There may be some testing code tweaks but I do not foresee any major changes from this point foreword other than passing flaky tests and/or some testing bugs. Due to `optimum` having the largest diffs and where most of the gptq quant logic is, we fill first concentrate on making sure `optimum` is review-cleared, then pef/transformers in that order. 

@SunMarc @MekkCyber  @Rocketknight1  This PR is ready and tested under cpu, intel/xpu, and nvidia/cuda. However, please note this PR is 1 of 3 in a cumulative set of 3 prs where the primary PR is in Optimum PR https://github.com/huggingface/optimum/pull/2064 (awaiting review/approval) while this and Peft  PR https://github.com/huggingface/peft/pull/2247 are dependent on getting the Optimum PR merged first. 

Can you also review these PRs @MekkCyber ?

Hi @SunMarc @MekkCyber . The [optimum PR](https://github.com/huggingface/optimum/pull/2064) has been merged. Please review this PR. Thanks!

@MekkCyber @SunMarc Also please trigger/approve the unit test so review can move forward. All tests passing on our end. Thanks. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35012). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @jiqing-feng, tests are not passing locally on my end, i'm having many errors related to `RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!`, could you verify please

@jiqing-feng @MekkCyber We have fix this bug and tests passing. Need to merged into this PR. 

passing:
* if only autogptq installed
* if only gptqmodel installed
* both gptqmodel and autogptq installed (logic should use gptqmodel as priority)

EDIT: changes may not be needed. 

Actually, the original optimum never support autogptq runs on CPU, optimum forced move the model to cuda in gptq quantizer. In my [new changes of optimum](https://github.com/huggingface/optimum/pull/2064), I'd like to remove the restirction that force model to cuda, but the maintainer refused. I think there is no need to test cpu when only autogptq available because the model will be forced move to cuda. I'd like to remove the CPU tests only when autogptq available if you agree.

@jiqing-feng Thanks for clearing my confusion on this as well. My previous comment about fix can be ignored 

GPTQModel has two kernels that can pass pure cpu based quantization and post-quant inference: Intel/Ipex [4bit] (avx, amx, xmx accelerated) + Torch [2, 3, 4, 8]. Autogptq has zero kernel that can perform on cpu.

So even if you ask model to use cpu only, optimum will, for autogptq, move model first layer to cuda and fail the cpu only test. 


@MekkCyber In light of this, can we petition to diable cpu testing with autogptq as jiqing-feng recommends? 


Gptqmodel has been cpu validated on amd zen3 zen4, intel xeon 4th-gen Sapphire Rapids, xeon 5th Emerald Rapids, and mac m1+. 



Refer to [this comment](https://github.com/huggingface/optimum/pull/2064#issuecomment-2550718167)

> Hey @jiqing-feng, tests are not passing locally on my end, i'm having many errors related to `RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!`, could you verify please

Could you please share your environment like torch version?

> Hey @jiqing-feng, tests are not passing locally on my end, i'm having many errors related to `RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!`, could you verify please

I've fixed it in [my new changes](https://github.com/huggingface/transformers/pull/35012/commits/409f6a2b5635afd28cca3b18cb735c6085242b94), please try again. Thanks!

I fixed it by moving the model to cuda when no gptqmodel was found so that the auto-gptq can run well with our [new merged PR](https://github.com/huggingface/optimum/pull/2064) on optimum!
(The error is mainly brought by our changes on optimum, we want to enable the model on CPU but optimum forced some tensors to cuda when only auto-gptq is available. We can fix it by also moving model to cuda when we found only auto-gptq available in transformers)

Hey @jiqing-feng thanks for the fix ! I am having some small fails related to `AssertionError: 'Hello my name is Katie. I am a 22 year' not found in {'Hello my name is Katie, I am a 22 year', 'Hello my name is Katie. I am a 20`, but these should be easy to fix maybe even after. For the cpu tests, when using `gptqmodel` the `test_serialization` is failling with `AssertionError: 'torch' != 'ipex'` any idea why ?

> Hey @jiqing-feng thanks for the fix ! I am having some small fails related to `AssertionError: 'Hello my name is Katie. I am a 22 year' not found in {'Hello my name is Katie, I am a 22 year', 'Hello my name is Katie. I am a 20`, but these should be easy to fix maybe even after. For the cpu tests, when using `gptqmodel` the `test_serialization` is failling with `AssertionError: 'torch' != 'ipex'` any idea why ?

Please run `pip install intel_extension_for_pytorch` and make sure your torch version is 2.5.0 or 2.5.1
Or run `pip install -v --no-build-isolation gptqmodel[ipex]`.

BTW, the result check is fixed. Please try again, thanks! 

Hi @MekkCyber . Please check my last comment and let me know if anything need to be changed before merging. Thanks!

> I think there is no need to test cpu when only autogptq available because the model will be forced move to cuda. I'd like to remove the CPU tests only when autogptq available if you agree.

I prefer to keep the cpu test for autogptq. Otherwise, there is no way to check that the per layer quantization works with auto-gptq. 

> Please run pip install intel_extension_for_pytorch and make sure your torch version is 2.5.0 or 2.5.1
> Or run pip install -v --no-build-isolation gptqmodel[ipex].

We shouldn't have to install intel_extension_for_pytorch to make the tests pass no ? 

> > I think there is no need to test cpu when only autogptq available because the model will be forced move to cuda. I'd like to remove the CPU tests only when autogptq available if you agree.
> 
> I prefer to keep the cpu test for autogptq. Otherwise, there is no way to check that the per layer quantization works with auto-gptq.

Yes, I have updated the test file to make sure all previous tests will be contained. But we found an error when testing device map, it should be fixed by the [tiny change of optimum](https://github.com/huggingface/optimum/pull/2136).

> > Please run pip install intel_extension_for_pytorch and make sure your torch version is 2.5.0 or 2.5.1
> > Or run pip install -v --no-build-isolation gptqmodel[ipex].
> 
> We shouldn't have to install intel_extension_for_pytorch to make the tests pass no ?

Right. Fixed in my new changes.

> Hey @jiqing-feng thanks for the fix ! I am having some small fails related to `AssertionError: 'Hello my name is Katie. I am a 22 year' not found in {'Hello my name is Katie, I am a 22 year', 'Hello my name is Katie. I am a 20`, but these should be easy to fix maybe even after. 

Based on our own testing, non-deterministic variability of the output is amplified in `cpu` testing. Looks like cpu implementation and execution of fp16 varies much more than `gpu`. Future testing code on `cpu` needs to take this into consideration. 

Thanks for the fix @jiqing-feng, @Qubitium, all tests are passing with both packages now 

@ArthurZucker @SunMarc @MekkCyber Anything else that is required of us to move this PR forward? Thanks!  We still have a lingering PEFT PR that is contingent on this PR be merged first.

Hey, super sorry reviewing in a bit I hope I was not the blocker ! 

Hi @SunMarc @ArthurZucker @MekkCyber . The [optimum PR](https://github.com/huggingface/optimum/pull/2136) has been merged, this PR should be ready to merge.

> Mostly wondering if it would not make more sense to create a separate backend as we can now treat them as different libs no? 🤗

For docs, we'd better put autogptq and gptqmodel in the same section because autogptq is no longer maintained, and we might deprecate autogptq in the future if it is incompatible.

> Mostly wondering if it would not make more sense to create a separate backend as we can now treat them as different libs no? 🤗 

To add to what @jiqing-feng mentioned. For this PR, backward compat and minimal code change on transformer/optimum/peft is target.

Deprecation is fully planned for autogptq with good reason.

Looking forward, long term, there is no reason to keep autogptq or to spend time doing arch and split hf gptq into two backends. The autogptq core maintainer has been mia, literally unreachable by anyone including @fxmarty (second maintainer) who did almost all the work in 2024 (until he ran out of spare time to work on it) when the project was active until I was invited by fxmarty to help as 3rd part-time restricted maintainer but I later decided to make GPTQModel instead in my vision unburdened by legacy api and what I considered questionable foundation code.


GPTQModel runs 100% ci feature and model coverage for each release. Autogptq has no ci. We have bugs too, but there are so many hidden bugs in autogptq we fixed we have lost count.

Transformers/Optimum/Peft only use the kernel part of autogptq code base so the full short/long term problems there are not visible here.




> Thanks for updating!

@stevhliu  Thanks for the doc/text corrections. 

I suppose this PR is ready to be merged as we got enough approvals, please let me know if anything I need to change.

Hi @ArthurZucker @Rocketknight1 . Pls let me know if there is anything I need to change before merging. Thx!

@sunmarc @mekkcyber I think you have the ultimate authority here. Take one last look and feel free to merge it if you're happy!

Sounds good ! I'll merge it then ! cc @MekkCyber for visibility