Package: auto-round
AutoRoundAdvanced Weight-Only Quantization Algorithm for LLMsAutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference. It's tailored for a wide range of models and consistently delivers noticeable improvements, often significantly outperforming SignRound with the cost of more tuning time for quantization.PrerequisitesPython 3.9 or higherInstallationBuild from Sourcepipinstall-rrequirements.txt
pythonsetup.pyinstallUsageOn CPU/GPUfromtransformersimportAutoModelForCausalLM,AutoTokenizerfromauto_roundimportAutoRoundmodel_name="meta-llama/Llama-2-7b-hf"model=AutoModelForCausalLM.from_pretrained(model_name,low_cpu_mem_usage=True,torch_dtype="auto",trust_remote_code=True)tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)bits,group_size,scheme=4,128,"asym"autoround=AutoRound(model,tokenizer,bits=bits,group_size=group_size,scheme=scheme)autoround.quantize()# Intel CPU Inference, For now only support llama, mistral and gpt-j.# then follow ITREX(https://github.com/intel/intel-extension-for-transformers/tree/main/intel_extension_for_transformers/llm/runtime/neural_speed) to load the model and do inference# currently please install neural-speed (https://github.com/intel/neural-speed) from sourceoutput_dir="./tmp_autoround"autoround.export(output_dir)fromintel_extension_for_transformers.transformersimportAutoModelForCausalLM,WeightOnlyQuantConfigwoq_config=WeightOnlyQuantConfig(group_size=group_size,scheme=scheme,use_autoround=True)##only supports 4 bits currentlyprompt="Once upon a time, a little girl"tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)inputs=tokenizer(prompt,return_tensors="pt").input_idsmodel=AutoModelForCausalLM.from_pretrained(model_name,quantization_config=woq_config,trust_remote_code=True,device="cpu")outputs=model.generate(inputs,max_new_tokens=30)Tuning on Intel Gaudi2fromtransformersimportAutoModelForCausalLM,AutoTokenizermodel_name="meta-llama/Llama-2-7b-hf"model=AutoModelForCausalLM.from_pretrained(model_name,low_cpu_mem_usage=True,torch_dtype="auto",trust_remote_code=True)tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)bits,group_size,scheme=4,128,"asym"# need to load model first, then importfromauto_roundimportAutoRoundautoround=AutoRound(model,tokenizer,bits=bits,group_size=group_size,scheme=scheme,device="hpu",scale_dtype="bf16",amp=False)autoround.quantize()Detailed Hyperparametersmodel: The PyTorch model to be quantized.tokenizer: An optional tokenizer for processing input data. If none is provided, a dataloader must be supplied.bits (int): Number of bits for quantization (default is 4).group_size (int): Size of the quantization group (default is 128).scheme (str): The quantization scheme (sym/asym) to be used (default is "asym").use_quant_input (bool): Whether to use the output of the previous quantized block as the input for the current block (default is True).enable_minmax_tuning (bool): Whether to enable weight min-max tuning (default is True).iters (int): Number of tuning iterations (default is 200).lr (float): The learning rate for rounding value (default is None, it will be set to 1.0/iters automatically).minmax_lr (float): The learning rate for min-max tuning (default is None, it will be set to lr automatically).n_samples (int): Number of samples for tuning (default is 512).seqlen (int): Data length of the sequence for tuning (default is 2048).bs (int): Batch size for training (default is 8).amp (bool): Whether to use automatic mixed precision (default is True).n_blocks (int): Packing several blocks as one for tuning together (default is 1).gradient_accumulate_steps (int): Number of gradient accumulation steps (default is 1).low_gpu_mem_usage (bool): Whether to save GPU memory at the cost of a little tuning time (default is True).dataset_name (str): The default dataset name for tuning (default is "NeelNanda/pile-10k").dataset_split (str): The split of the dataset to be used for tuning (default is "train").dataloader: The dataloader for tuning data.weight_config (dict): Configuration for weight quantization (default is an empty dictionary), mainly for mixed bits or mixed precision.device: The device to be used for tuning (default is "cuda:0").Validated ModelsFor wikitext2/ptb-new/c4-new ppl, we follow the code of gptq and set the sequence length to 2048. For lm-eval wikitext ppl, we adopt lm-eval. The quantization configure is W4G128.ModelMethodAcc AVG.MMLULamb.Hella.Wino.PiqaTruth.Open.BoolqRTEARC-eARC-c.wikitext2 pplptb_new pplc4_new ppllm_eval wikitext pplIntel/neural-chat-7b-v3FP1667.9261.1373.0366.3976.4081.0147.3738.886.9775.8182.6657.516.0048.969.65-Ours66.9060.5672.1965.2875.3781.1846.7636.086.9173.2981.7356.666.2159.7810.01-Ours iters=1K,use_quant_input=False, minmax_lr=0.00267.7060.5773.7465.6277.4380.8547.6136.886.9475.0982.6657.346.1759.129.83-mistralai/Mixtral-8x7B-v0.1BF1667.1669.8378.4464.8976.4082.4334.1535.4084.9871.1284.2256.913.8419.227.41-Ours65.9868.9078.1164.3174.2782.1030.9734.2084.5767.8783.9656.574.083547.56-Ours iters=1K,use_quant_input=False66.7868.6878.6164.4076.5681.9932.5634.8085.9670.7683.9656.313.9917.657.52-microsoft/phi-2FP1661.8056.4062.7855.8375.7778.6731.2140.4083.3662.4580.0552.909.7118.1614.1211.05Ours61.6754.5761.3255.0476.4878.8929.7440.6083.2466.4379.7652.309.9818.6714.3911.37Ours iters=1K,use_quant_input=False61.4755.4161.7754.9276.4078.2931.0940.083.2463.5479.2952.229.9718.6314.3711.35We provide acomprehensive analysiswith other methods in our accuracy data section. Notably, our approach has outperformed GPTQ with a score of 30/32 and AWQ with a score of 27/32 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, W2G128.  And the tuning costs are comparable.Tips1 Consider increasing tuning steps to achieve better results, albeit with increased tuning time. Additionally, setting 'use_quant_input' to False or adjusting 'minmax_lr' to 2.0/iters has been observed to occasionally yield improved results.2 Leverage AutoGPTQ to run the model on GPUfromtransformersimportAutoModelForCausalLM,AutoTokenizerfromauto_roundimportAutoRoundmodel_name="facebook/opt-125m"model=AutoModelForCausalLM.from_pretrained(model_name,low_cpu_mem_usage=True,torch_dtype="auto",trust_remote_code=True)tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)autoround=AutoRound(model,tokenizer,bits=4,group_size=128,scheme="asym")autoround.quantize()## export to autogptq# please install auto-gptq https://github.com/AutoGPTQ/output_dir="/path/to/quantized_model"autoround.export(output_dir,target="auto_gptq",use_triton=True)# then follow auto-gptq to load the model and inferenceExamplesQuantization has been enabled for various large language models. Please refer to theexample readmefor details.ReferenceIf you find SignRound useful for your research, please cite our paper:@article{cheng2023optimize,title={OptimizeWeightRoundingviaSignedGradientDescentfortheQuantizationofLLMs},author={Cheng,WenhuaandZhang,WeiweiandShen,HaihaoandCai,YiyangandHe,XinandLv,Kaokao},journal={arXivpreprintarXiv:2309.05516},year={2023}}