0-r4-web.pdf) . However, to the best of our knowledge, there are
no open-source W4A8 quantization kernels available, but this may well be [ an
interesting direction to explore
](https://www.qualcomm.com/news/onq/2023/04/floating-point-arithmetic-for-ai-
inference-hit-or-miss) .

On the kernel side as well, designing performant W4A16 kernels for larger
batch sizes remains an open challenge.

###  Supported models

In this initial implementation, only large language models with a decoder or
encoder only architecture are supported. This may sound a bit restrictive, but
it encompasses most state of the art LLMs such as Llama, OPT, GPT-Neo, GPT-
NeoX.

Very large vision, audio, and multi-modal models are currently not supported.

##  Conclusion and final words

In this blogpost we have presented the integration of the [ AutoGPTQ library
](https://github.com/PanQiWei/AutoGPTQ) in Transformers, making it possible to
quantize LLMs with the GPTQ method to make them more accessible for anyone in
the community and empower them to build exciting tools and applications with
LLMs.

This integration is available both for Nvidia GPUs, and RoCm-powered AMD GPUs,
which is a huge step towards democratizing quantized models for broader GPU
architectures.

The collaboration with the AutoGPTQ team has been very fruitful, and we are
very grateful for their support and their work on this library.

We hope that this integration will make it easier for everyone to use LLMs in
their applications, and we are looking forward to seeing what you will build
with it!

Do not miss the useful resources shared above for better understanding the
integration and how to quickly get started with GPTQ quantization.

  * [ Original Paper ](https://arxiv.org/pdf/2210.17323.pdf)
  * [ Basic usage Google Colab notebook ](https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb?usp=sharing) \- This notebook shows how to quantize your transformers model with GPTQ method, how to do inference, and how to do fine-tuning with the quantized model. 
  * Transformers integration [ documentation ](https://huggingface.co/docs/transformers/main/en/main_classes/quantization)
  * Optimum integration [ documentation ](https://huggingface.co/docs/optimum/llm_quantization/usage_guides/quantization)
  * The Bloke [ repositories ](https://huggingface.co/TheBloke?sort_models=likes#models) with compatible GPTQ models. 

##  Acknowledgements

We would like to thank [ William ](https://github.com/PanQiWei) for his
support and his work on the amazing AutoGPTQ library and for his help in the
integration. We would also like to thank [ TheBloke
](https://huggingface.co/TheBloke) for his work on quantizing many models with
AutoGPTQ and sharing them on the Hub and for his help with the integration. We
would also like to aknowledge [ qwopqwop200 ](https://github.com/qwopqwop200)
for his continuous contributions on AutoGPTQ library and his work on extending
the library for CPU that is going to be released in the next versions of
AutoGPTQ.

Finally, we would like to thank [ Pedro Cuenca ](https://github.com/pcuenca)
for his help with the writing of this blogpost.

More articles from our Blog

[ ![](/blog/assets/cloudflare-workers-ai/thumbnail.jpg) Bringing serverless
GPU inference to Hugging Face users  By  [ philschmid ](/philschmid) April 2,
2024  ](/blog/cloudflare-workers-ai) [ ![](/blog/assets/phi2-intel-meteor-
lake/02.jpg) A Chatbot on your Laptop: Phi-2 on Intel Meteor Lake  By  [
juliensimon ](/juliensimon) March 20, 2024  ](/blog/phi2-intel-meteor-lake)

Company

© Hugging Face

[ TOS ](/terms-of-service) [ Privacy ](/privacy) [ About ](/huggingface) [
Jobs ](https://apply.workable.com/huggingface/) [ ](/)

Website

[ Models ](/models) [ Datasets ](/datasets) [ Spaces ](/spaces) [ Pricing
](/pricing) [ Docs ](/docs)