Package: qllm
QLLMSupports any LLMs in HuggingFace/Transformers, mixed bits(2-8bit), GPTQ/AWQ/HQQ, ONNX exportQLLM is a out-of-box quantization toolbox for large language models, It didn't limit to a specific model, and designed to be auto-quantization layer by layer for any LLMs. It can also be used to export quantized model to onnx with only one args `--export_onnx ./onnx_model`, and inference with onnxruntime.
Besides, model quantized by different quantization method (GPTQ/AWQ/HQQ) can be loaded from huggingface/transformers and transfor to each other without extra effort.We alread supportedGPTQ quantizationAWQ quantizationHQQ quantizationFeatures:GPTQ supports all LLM models in huggingface/transformers, it will automatically detect the model type and quantize it.We support to quantize model by 2-8 bits, and support to quantize model with different quantization bits for different layers.Auto promoting bits/group-size for better accuracyLatest News🔥[2024/01] SupportHQQalgorithm[2023/12] The first PyPi package releasedInstallationEasy to install qllm from PyPi [cu121]pip install qllmInstall from release package, CUDA-118/121 is supported.
[py38, py39, py310]https://github.com/wejoncy/QLLM/releasesBuild from SourcePlease set ENV EXCLUDE_EXTENTION_FOR_FAST_BUILD=1 for fast buildIf you are using CUDA-121pip install git+https://github.com/wejoncy/QLLM.gitOR CUDA-118/117git clone https://github.com/wejoncy/QLLM.git
cd QLLM
python setup.py installHow to use itQuantize llama2#  Quantize and Save compressed model, method can be one of [gptq/awq/hqq]python-mqllm--model=meta-llama/Llama-2-7b-hf--method=gptq--nsamples=64--wbits=4--groupsize=128--save./Llama-2-7b-4bit(NEW) Quantize model with mix bits/groupsize for higher precision (PPL)#  Quantize and Save compressed modelpython-mqllm--model=meta-llama/Llama-2-7b-hf--method=gptq--save./Llama-2-7b-4bit--allow_mix_bits--true-sequentialNOTE:only support GPTQallow_mix_bits option refered from gptq-for-llama, QLLM makes it easier to use and flexiblewjat different with gptq-for-llama is we grow bit by one instead of times 2.all configurations will be saved/load automaticlly instead of quant-table which used by gptq-for-llama.if --allow_mix_bits is enabled, The saved model is not compatible with vLLM for now.Quantize model for vLLMDue to the zereos diff, we need to set a env variable if you set pack_mode to GPTQ whenver the method is awq or gptqCOMPATIBLE_WITH_AUTOGPTQ=1python-mqllm--model=meta-llama/Llama-2-7b-hf--method=gptq--save./Llama-2-7b-4bit--pack_mode=GPTQIf you use GEMM pack_mode, then you don't have to set the varpython-mqllm--model=meta-llama/Llama-2-7b-hf--method=gptq--save./Llama-2-7b-4bit--pack_mode=GEMMpython-mqllm--model=meta-llama/Llama-2-7b-hf--method=awq--save./Llama-2-7b-4bit--pack_mode=GEMMConversion between AWQ and GPTQpython-mqllm--loadTheBloke/Llama-2-7B-Chat-AWQ--eval--save./Llama-2-7b-chat-hf_gptq_q4/--pack_mode=GPTQOr you can use--pack_mode=AWQto convert GPTQ to AWQ.python-mqllm--loadTheBloke/Llama-2-7B-Chat-GPTQ--eval--save./Llama-2-7b-chat-hf_awq_q4/--pack_mode=GEMMNote:Not all cases are supported, for example,if you quantized model with different quantization bits for different layers, you can't convert it to AWQ.if GPTQ model is quantized with--allow_mix_bitsoption, you can't convert it to AWQ.if GPTQ model is quantized with--act_orderoption, you can't convert it to AWQ.Convert to onnx modeluse--export_onnx ./onnx_modelto export and save onnx modelpython -m qllm --model  meta-llama/Llama-2-7b-chat-hf  --method=gptq  --dataset=pileval --nsamples=16  --save ./Llama-2-7b-chat-hf_awq_q4/ --export_onnx ./Llama-2-7b-chat-hf_awq_q4_onnx/model inference with the saved modelpython-mqllm--load./Llama-2-7b-4bit--evalmodel inference with ORTyou may want to usegenaito do generation with ORT.importonnxruntimefromtransformersimportAutoTokenizeronnx_path_str='./Llama-2-7b-4bit-onnx'tokenizer=AutoTokenizer.from_pretrained(onnx_path_str,use_fast=True)sample_inputs=tokenizer("Hello, my dog is cute",return_tensors="pt")onnx_model_path=onnx_path_str+'/decoder_merged.onnx'session=onnxruntime.InferenceSession(onnx_model_path,providers=['CUDAExecutionProvider'])mask=np.ones(sample_inputs[0].shape,dtype=np.int64)ifsample_inputs[1]isNoneelsesample_inputs[1].cpu().numpy()num_layers=model.config.num_hidden_layersinputs={'input_ids':sample_inputs[0].cpu().numpy(),'attention_mask':mask,'position_ids':np.arrange(0,sample_inputs[0],dtype=np.int64),'use_cache_branch':np.array([0],dtype=np.bool_)}foriinrange(num_layers):inputs[f'past_key_values.{i}.key']=np.zeros((1,32,32,128),dtype=np.float16)inputs[f'past_key_values.{i}.value']=np.zeros((1,32,32,128),dtype=np.float16)outputs=session(None,inputs)Load quantized model from hugingface/transformerspython-mqllm--loadTheBloke/Llama-2-7B-Chat-AWQ--eval
python-mqllm--loadTheBloke/Llama-2-7B-Chat-GPTQ--eval
python-mqllm--loadTheBloke/Mixtral-8x7B-v0.1-GPTQ--use_pluginstart a chatbotyou may need to install fschat and accelerate with pippipinstallfschataccelerateuse--use_pluginto enable a chatbot pluginpython -m qllm --model  meta-llama/Llama-2-7b-chat-hf  --method=awq  --dataset=pileval --nsamples=16  --use_plugin --save ./Llama-2-7b-chat-hf_awq_q4/

or 
python -m qllm --model  meta-llama/Llama-2-7b-chat-hf  --method=gptq  --dataset=pileval --nsamples=16  --use_plugin --save ./Llama-2-7b-chat-hf_gptq_q4/For some users has transformers connect issues.Please set environment with PROXY_PORT=your http proxy portPowerShell$env:PROXY_PORT=1080Bashexport PROXY_PORT=1080windows cmdset PROXY_PORT=1080AcknowledgementsGPTQGPTQ-tritonAutoGPTQllm-awqAutoAWQ.HQQ