Package: auto-gptq
AutoGPTQAn easy-to-use LLM quantization package with user-friendly APIs, based on GPTQ algorithm (weight-only quantization).English|中文News or Update2024-02-15 - (News) - AutoGPTQ 0.7.0 is released, withMarlinint4*fp16 matrix multiplication kernel support, with the argumentuse_marlin=Truewhen loading models.2023-08-23 - (News) - 🤗 Transformers, optimum and peft have integratedauto-gptq, so now running and training GPTQ models can be more available to everyone! Seethis blogand it's resources for more details!For more histories please turn toherePerformance ComparisonInference SpeedThe result is generated usingthis script, batch size of input is 1, decode strategy is beam search and enforce the model to generate 512 tokens, speed metric is tokens/s (the larger, the better).The quantized model is loaded using the setup that can gain the fastest inference speed.modelGPUnum_beamsfp16gptq-int4llama-7b1xA100-40G118.8725.53llama-7b1xA100-40G468.7991.30moss-moon 16b1xA100-40G112.4815.25moss-moon 16b1xA100-40G4OOM42.67moss-moon 16b2xA100-40G106.8306.78moss-moon 16b2xA100-40G413.1010.80gpt-j 6b1xRTX3060-12G1OOM29.55gpt-j 6b1xRTX3060-12G4OOM47.36PerplexityFor perplexity comparison, you can turn tohereandhereInstallationAutoGPTQ is available on Linux and Windows only. You can install the latest stable release of AutoGPTQ from pip with pre-built wheels:CUDA/ROCm versionInstallationBuilt against PyTorchCUDA 11.8pip install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/2.2.0+cu118CUDA 12.1pip install auto-gptq2.2.0+cu121ROCm 5.7pip install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/rocm573/2.2.0+rocm5.7AutoGPTQ can be installed with the Triton dependency withpip install auto-gptq[triton]in order to be able to use the Triton backend (currently only supports linux, no 3-bits quantization).For older AutoGPTQ, please refer tothe previous releases installation table.Install from sourceClone the source code:gitclonehttps://github.com/PanQiWei/AutoGPTQ.git&&cdAutoGPTQA few packages are required in order to build from source:pip install numpy gekko pandas.Then, install locally from source:pipinstall-vvv-e.You can setBUILD_CUDA_EXT=0to disable pytorch extension building, but this isstrongly discouragedas AutoGPTQ then falls back on a slow python implementation.On ROCm systemsTo install from source for AMD GPUs supporting ROCm, please specify theROCM_VERSIONenvironment variable. Example:ROCM_VERSION=5.6pipinstall-vvv-e.The compilation can be speeded up by specifying thePYTORCH_ROCM_ARCHvariable (reference) in order to build for a single target device, for examplegfx90afor MI200 series devices.For ROCm systems, the packagesrocsparse-dev,hipsparse-dev,rocthrust-dev,rocblas-devandhipblas-devare required to build.Quick TourQuantization and Inferencewarning: this is just a showcase of the usage of basic apis in AutoGPTQ, which uses only one sample to quantize a much small model, quality of quantized model using such little samples may not good.Below is an example for the simplest use ofauto_gptqto quantize a model and inference after quantization:fromtransformersimportAutoTokenizer,TextGenerationPipelinefromauto_gptqimportAutoGPTQForCausalLM,BaseQuantizeConfigimportlogginglogging.basicConfig(format="%(asctime)s%(levelname)s[%(name)s]%(message)s",level=logging.INFO,datefmt="%Y-%m-%d%H:%M:%S")pretrained_model_dir="facebook/opt-125m"quantized_model_dir="opt-125m-4bit"tokenizer=AutoTokenizer.from_pretrained(pretrained_model_dir,use_fast=True)examples=[tokenizer("auto-gptq is an easy-to-use model quantization library with user-friendly apis, based on GPTQ algorithm.")]quantize_config=BaseQuantizeConfig(bits=4,# quantize model to 4-bitgroup_size=128,# it is recommended to set the value to 128desc_act=False,# set to False can significantly speed up inference but the perplexity may slightly bad)# load un-quantized model, by default, the model will always be loaded into CPU memorymodel=AutoGPTQForCausalLM.from_pretrained(pretrained_model_dir,quantize_config)# quantize model, the examples should be list of dict whose keys can only be "input_ids" and "attention_mask"model.quantize(examples)# save quantized modelmodel.save_quantized(quantized_model_dir)# save quantized model using safetensorsmodel.save_quantized(quantized_model_dir,use_safetensors=True)# push quantized model to Hugging Face Hub.# to use use_auth_token=True, Login first via huggingface-cli login.# or pass explcit token with: use_auth_token="hf_xxxxxxx"# (uncomment the following three lines to enable this feature)# repo_id = f"YourUserName/{quantized_model_dir}"# commit_message = f"AutoGPTQ model for {pretrained_model_dir}: {quantize_config.bits}bits, gr{quantize_config.group_size}, desc_act={quantize_config.desc_act}"# model.push_to_hub(repo_id, commit_message=commit_message, use_auth_token=True)# alternatively you can save and push at the same time# (uncomment the following three lines to enable this feature)# repo_id = f"YourUserName/{quantized_model_dir}"# commit_message = f"AutoGPTQ model for {pretrained_model_dir}: {quantize_config.bits}bits, gr{quantize_config.group_size}, desc_act={quantize_config.desc_act}"# model.push_to_hub(repo_id, save_dir=quantized_model_dir, use_safetensors=True, commit_message=commit_message, use_auth_token=True)# load quantized model to the first GPUmodel=AutoGPTQForCausalLM.from_quantized(quantized_model_dir,device="cuda:0")# download quantized model from Hugging Face Hub and load to the first GPU# model = AutoGPTQForCausalLM.from_quantized(repo_id, device="cuda:0", use_safetensors=True, use_triton=False)# inference with model.generateprint(tokenizer.decode(model.generate(**tokenizer("auto_gptq is",return_tensors="pt").to(model.device))[0]))# or you can also use pipelinepipeline=TextGenerationPipeline(model=model,tokenizer=tokenizer)print(pipeline("auto-gptq is")[0]["generated_text"])For more advanced features of model quantization, please reference tothis scriptCustomize ModelBelow is an example to extend `auto_gptq` to support `OPT` model, as you will see, it's very easy:fromauto_gptq.modelingimportBaseGPTQForCausalLMclassOPTGPTQForCausalLM(BaseGPTQForCausalLM):# chained attribute name of transformer layer blocklayers_block_name="model.decoder.layers"# chained attribute names of other nn modules that in the same level as the transformer layer blockoutside_layer_modules=["model.decoder.embed_tokens","model.decoder.embed_positions","model.decoder.project_out","model.decoder.project_in","model.decoder.final_layer_norm"]# chained attribute names of linear layers in transformer layer module# normally, there are four sub lists, for each one the modules in it can be seen as one operation,# and the order should be the order when they are truly executed, in this case (and usually in most cases),# they are: attention q_k_v projection, attention output projection, MLP project input, MLP project outputinside_layer_modules=[["self_attn.k_proj","self_attn.v_proj","self_attn.q_proj"],["self_attn.out_proj"],["fc1"],["fc2"]]After this, you can useOPTGPTQForCausalLM.from_pretrainedand other methods as shown in Basic.Evaluation on Downstream TasksYou can use tasks defined inauto_gptq.eval_tasksto evaluate model's performance on specific down-stream task before and after quantization.The predefined tasks support all causal-language-models implemented in🤗 transformersand in this project.Below is an example to evaluate `EleutherAI/gpt-j-6b` on sequence-classification task using `cardiffnlp/tweet_sentiment_multilingual` dataset:fromfunctoolsimportpartialimportdatasetsfromtransformersimportAutoTokenizer,AutoModelForCausalLM,GenerationConfigfromauto_gptqimportAutoGPTQForCausalLM,BaseQuantizeConfigfromauto_gptq.eval_tasksimportSequenceClassificationTaskMODEL="EleutherAI/gpt-j-6b"DATASET="cardiffnlp/tweet_sentiment_multilingual"TEMPLATE="Question:What's the sentiment of the given text? Choices are{labels}.\nText:{text}\nAnswer:"ID2LABEL={0:"negative",1:"neutral",2:"positive"}LABELS=list(ID2LABEL.values())defds_refactor_fn(samples):text_data=samples["text"]label_data=samples["label"]new_samples={"prompt":[],"label":[]}fortext,labelinzip(text_data,label_data):prompt=TEMPLATE.format(labels=LABELS,text=text)new_samples["prompt"].append(prompt)new_samples["label"].append(ID2LABEL[label])returnnew_samples#  model = AutoModelForCausalLM.from_pretrained(MODEL).eval().half().to("cuda:0")model=AutoGPTQForCausalLM.from_pretrained(MODEL,BaseQuantizeConfig())tokenizer=AutoTokenizer.from_pretrained(MODEL)task=SequenceClassificationTask(model=model,tokenizer=tokenizer,classes=LABELS,data_name_or_path=DATASET,prompt_col_name="prompt",label_col_name="label",**{"num_samples":1000,# how many samples will be sampled to evaluation"sample_max_len":1024,# max tokens for each sample"block_max_len":2048,# max tokens for each data block# function to load dataset, one must only accept data_name_or_path as input# and return datasets.Dataset"load_fn":partial(datasets.load_dataset,name="english"),# function to preprocess dataset, which is used for datasets.Dataset.map,# must return Dict[str, list] with only two keys: [prompt_col_name, label_col_name]"preprocess_fn":ds_refactor_fn,# truncate label when sample's length exceed sample_max_len"truncate_prompt":False})# note that max_new_tokens will be automatically specified internally based on given classesprint(task.run())# self-consistencyprint(task.run(generation_config=GenerationConfig(num_beams=3,num_return_sequences=3,do_sample=True)))Learn Moretutorialsprovide step-by-step guidance to integrateauto_gptqwith your own project and some best practice principles.examplesprovide plenty of example scripts to useauto_gptqin different ways.Supported Modelsyou can usemodel.config.model_typeto compare with the table below to check whether the model you use is supported byauto_gptq.for example, model_type ofWizardLM,vicunaandgpt4allare allllama, hence they are all supported byauto_gptq.model typequantizationinferencepeft-lorapeft-ada-lorapeft-adaption_promptbloom✅✅✅✅gpt2✅✅✅✅gpt_neox✅✅✅✅✅requires this peft branchgptj✅✅✅✅✅requires this peft branchllama✅✅✅✅✅moss✅✅✅✅✅requires this peft branchopt✅✅✅✅gpt_bigcode✅✅✅✅codegen✅✅✅✅falcon(RefinedWebModel/RefinedWeb)✅✅✅✅Supported Evaluation TasksCurrently,auto_gptqsupports:LanguageModelingTask,SequenceClassificationTaskandTextSummarizationTask; more Tasks will come soon!Running testsTests can be run with:pytest tests/ -sFAQWhich kernel is used by default?AutoGPTQ defaults to using exllamav2 int4*fp16 kernel for matrix multiplication.How to use Marlin kernel?Marlin is an optimized int4 * fp16 kernel was recently proposed athttps://github.com/IST-DASLab/marlin. This is integrated in AutoGPTQ when loading a model withuse_marlin=True. This kernel is available only on devices with compute capability 8.0 or 8.6 (Ampere GPUs).AcknowledgementSpecial thanksElias Frantar,Saleh Ashkboos,Torsten HoeflerandDan Alistarhfor proposingGPTQalgorithm and open source thecode, and for releasingMarlin kernelfor mixed precision computation.Special thanksqwopqwop200, for code in this project that relevant to quantization are mainly referenced fromGPTQ-for-LLaMa.Special thanks toturboderp, for releasingExllamaandExllama v2libraries with efficient mixed precision kernels.