[ ![Hugging Face's logo](/front/assets/huggingface_logo-noborder.svg) Hugging
Face  ](/)

  * [ Models ](/models)
  * [ Datasets ](/datasets)
  * [ Spaces ](/spaces)
  * [ Posts ](/posts)
  * [ Docs ](/docs)
  * Solutions 

  * [ Pricing ](/pricing)
  *   * * * *

  * [ Log In ](/login)
  * [ Sign Up ](/join)

[ Back to blog ](/blog)

#  Making LLMs lighter with AutoGPTQ and transformers

Published August 23, 2023

[ Update on GitHub ](https://github.com/huggingface/blog/blob/main/gptq-
integration.md)

[ ![Marc Sun's avatar](https://aeiljuispo.cloudimg.io/v7/https://cdn-
uploads.huggingface.co/production/uploads/63ce875d199b36f7552d4f07/NrTlnhUTAp0j0ZsMTsWIu.jpeg?w=200&h=200&f=face)
marcsun13  Marc Sun  ](/marcsun13) [ ![Félix Marty's
avatar](https://aeiljuispo.cloudimg.io/v7/https://cdn-
uploads.huggingface.co/production/uploads/1651743336129-624c60cba8ec93a7ac188b56.png?w=200&h=200&f=face)
fxmarty  Félix Marty  ](/fxmarty) [ ![潘其威's
avatar](/avatars/ad2b3f618031a1ea695728053a714fc2.svg) PanEa  潘其威  guest
](/PanEa) [ ![Junjae Lee's
avatar](/avatars/1bcbc93e7343b8dde06601abf6e32f4e.svg) qwopqwop  Junjae Lee
guest  ](/qwopqwop) [ ![Younes Belkada's
avatar](https://aeiljuispo.cloudimg.io/v7/https://cdn-
uploads.huggingface.co/production/uploads/1648631057413-noauth.png?w=200&h=200&f=face)
ybelkada  Younes Belkada  ](/ybelkada) [ ![Tom Jobbins's
avatar](https://aeiljuispo.cloudimg.io/v7/https://cdn-
uploads.huggingface.co/production/uploads/6426d3f3a7723d62b53c259b/tvPikpAzKTKGN5wrpadOJ.jpeg?w=200&h=200&f=face)
TheBloke  Tom Jobbins  guest  ](/TheBloke)

  * Table of contents 
  * Resources 
  * **A gentle summary of the GPTQ paper**
  * AutoGPTQ library – the one-stop library for efficiently leveraging GPTQ for LLMs 
  * Native support of GPTQ models in 🤗 Transformers 
  * Quantizing models **with the Optimum library**
  * Running GPTQ models through _**Text-Generation-Inference** _
  * **Fine-tune quantized models with PEFT**
  * Room for improvement 
    * Supported models 
  * Conclusion and final words 
  * Acknowledgements 

This article is also available in Chinese [ 简体中文 ](/blog/zh/gptq-integration)
.

Large language models have demonstrated remarkable capabilities in
understanding and generating human-like text, revolutionizing applications
across various domains. However, the demands they place on consumer hardware
for training and deployment have become increasingly challenging to meet.

🤗 Hugging Face's core mission is to _democratize good machine learning_ , and
this includes making large models as accessible as possible for everyone. In
the same spirit as our [ bitsandbytes collaboration
](https://huggingface.co/blog/4bit-transformers-bitsandbytes) , we have just
integrated the [ AutoGPTQ ](https://github.com/PanQiWei/AutoGPTQ) library in
Transformers, making it possible for users to quantize and run models in 8, 4,
3, or even 2-bit precision using the GPTQ algorithm ( [ Frantar et al. 2023
](https://arxiv.org/pdf/2210.17323.pdf) ). There is negligible accuracy
degradation with 4-bit quantization, with inference speed comparable to the `
fp16 ` baseline for small batch sizes. Note that GPTQ method slightly differs
from post-training quantization methods proposed by bitsandbytes as it
requires to pass a calibration dataset.

This integration is available both for Nvidia GPUs, and RoCm-powered AMD GPUs.

##  Table of contents

  * Resources 
  * **A gentle summary of the GPTQ paper**
  * AutoGPTQ library – the one-stop library for efficiently leveraging GPTQ for LLMs 
  * Native support of GPTQ models in 🤗 Transformers 
  * Quantizing models **with the Optimum library**
  * Running GPTQ models through _**Text-Generation-Inference** _
  * **Fine-tune quantized models with PEFT**
  * Room for improvement 
    * Supported models 
  * Conclusion and final words 
  * Acknowledgements 

##  Resources

This blogpost and release come with several resources to get started with GPTQ
quantization:

  * [ Original Paper ](https: