Package: auto-around
AutoRoundAdvanced Weight-Only Quantization Algorithm for LLMsAutoRound is an advanced weight-only quantization algorithm, based on SignRound. It's tailored for a wide range of models and consistently delivers noticeable improvements, often significantly outperforming SignRound with the cost of more tuning time for quantization.PrerequisitesPython 3.9 or higherInstallationBuild from Sourcepipinstall-rrequirements.txt
pythonsetup.pyinstallUsagefromtransformersimportAutoModelForCausalLM,AutoTokenizerfromauto_roundimportAutoRoundmodel_name="facebook/opt-125m"model=AutoModelForCausalLM.from_pretrained(model_name,low_cpu_mem_usage=True,torch_dtype="auto",trust_remote_code=True)tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)bits,group_size,scheme=4,128,"asym"autoround=AutoRound(model,tokenizer,bits=bits,group_size=group_size,scheme=scheme)autoround.quantize()Detailed Hyperparametersmodel: The PyTorch model to be quantized.tokenizer: An optional tokenizer for processing input data. If none is provided, a dataloader must be supplied.bits (int): Number of bits for quantization (default is 4).group_size (int): Size of the quantization group (default is 128).scheme (str): The quantization scheme (symmetric/asymmetric) to be used (default is "asym").use_quant_input (bool): Whether to use the output of the previous quantized block as the input for the current block (default is True).enable_minmax_tuning (bool): Whether to enable weight min-max tuning (default is True).iters (int): Number of tuning iterations (default is 200).lr (float): The learning rate for rounding value (default is None, it will be set to 1.0/iters automatically).minmax_lr (float): The learning rate for min-max tuning (default is None, it will be set to lr automatically).n_samples (int): Number of samples for tuning (default is 512).seqlen (int): Data length of the sequence for tuning (default is 2048).bs (int): Batch size for training (default is 8).amp (bool): Whether to use automatic mixed precision (default is True).n_blocks (int): Packing several blocks as one for tuning together (default is 1).gradient_accumulate_steps (int): Number of gradient accumulation steps (default is 1).low_gpu_mem_usage (bool): Whether to save GPU memory at the cost of a little tuning time (default is True).dataset_name (str): The default dataset name for tuning (default is "NeelNanda/pile-10k").dataset_split (str): The split of the dataset to be used for tuning (default is "train").dataloader: The dataloader for tuning data.weight_config (dict): Configuration for weight quantization (default is an empty dictionary), mainly for mixed bits or mixed precision.device: The device to be used for tuning (default is "cuda:0").Validated ModelsFor wikitext2/ptb-new/c4-new ppl, we follow the code of gptq and set the sequence length to 2048. For lm-eval wikitext ppl, we adopt lm-eval. The quantization configure is W4G128.ModelMethodAcc AVG.MMLULamb.Hella.Wino.PiqaTruth.Open.BoolqRTEARC-eARC-c.wikitext2 pplptb_new pplc4_new ppllm_eval wikitext pplIntel/neural-chat-7b-v3FP1667.9261.1373.0366.3976.4081.0147.3738.886.9775.8182.6657.516.0048.969.65-Ours66.9060.5672.1965.2875.3781.1846.7636.086.9173.2981.7356.666.2159.7810.01-Ours iters1K, disable use_quant_input, minmax_lr 0.00267.7060.5773.7465.6277.4380.8547.6136.886.9475.0982.6657.346.1759.129.83-mistralai/Mixtral-8x7B-v0.1BF1667.1669.8378.4464.8976.4082.4334.1535.4084.9871.1284.2256.913.8419.227.41-Ours65.9868.9078.1164.3174.2782.1030.9734.2084.5767.8783.9656.574.083547.56-Ours iters1K, disable use_quant_input66.7868.6878.6164.4076.5681.9932.5634.8085.9670.7683.9656.313.9917.657.52-microsoft/phi-2FP1661.8056.4062.7855.8375.7778.6731.2140.4083.3662.4580.0552.909.7118.1614.1211.05AutoRound61.6754.5761.3255.0476.4878.8929.7440.6083.2466.4379.7652.309.9818.6714.3911.37We provide a comparative analysis with other methodslinkin our accuracy data section. Notably, our approach has outperformed GPTQ with a score of 30/32 and AWQ with a score of 27/32 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, W2G128.  And the tuning costs are comparable.Models passed smoke testLaMini-GPT-124M; QWEN1-8B; OPT-125M; Bloom-560m;falcon-7b;gpt-leo-125m;stablelm-base-alpha-3b;dolly-v2-3b;mpt-7b;gpt-j-6b;chatglm2-6bTips1 Consider increasing tuning steps to achieve better results, albeit with increased tuning time.2 Leverage AutoGPTQ to evaluate the model on GPUfromtransformersimportAutoModelForCausalLM,AutoTokenizerfromauto_roundimportAutoRoundmodel_name="facebook/opt-125m"model=AutoModelForCausalLM.from_pretrained(model_name,low_cpu_mem_usage=True,torch_dtype="auto",trust_remote_code=True)tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)autoround=AutoRound(model,tokenizer,bits=4,group_size=128,scheme="asym")autoround.quantize()## export to autogptq# please install auto-gptq https://github.com/AutoGPTQ/output_dir="/path/to/quantized_model"autoround.export(output_dir,target="auto_gptq",use_triton=True)# then follow auto-gptq to load the model and inferenceKnown IssuesRandom issues in tuning Qwen modelsChatGlm-V1 is not supportedExamplesEnter into the examples folder and install lm-eval to run the evaluationpipinstall-rrequirements.txtDefault Settings:CUDA_VISIBLE_DEVICES=0python3main.py--model_namefacebook/opt-125m--amp--bits4--group_size-1--enable_minmax_tuning--use_quant_inputReduced GPU Memory Usage and Adjusted Training Batch Size:CUDA_VISIBLE_DEVICES=0python3main.py--model_namefacebook/opt-125m--amp--bits4--group_size-1--low_gpu_mem_usage--train_bs1--gradient_accumulate_steps8Utilizing the AdamW Optimizer:Include the flag--adam. Note that AdamW is less effective than Sign gradient descent in many scenarios we tested.Running the Original SignRound:CUDA_VISIBLE_DEVICES=0python3main.py--model_namefacebook/opt-125m--amp--bits4--group_size-1--iters400--lr0.0025--minmax_lr0.0025--enable_minmax_tuningis strongly recommendedThe transformers version required varies across different types of models. Here, the transformers version used for running models during experiments is provided as a reference.ModelTransformers versionEleutherAI/gpt-j-6b4.28/4.30/4.34/4.36huggyllama/llama-7b4.28/4.30/4.34/4.36meta-llama/Llama-2-7b-hf4.30/4.34/4.36facebook/opt-6.7b4.28/4.30/4.34/4.36tiiuae/falcon-7b4.28/4.30/4.34/4.36mosaicml/mpt-7b4.28/4.30/4.34/4.36bigscience/bloom-7b14.28/4.30/4.34/4.36baichuan-inc/Baichuan-7B4.28/4.30Qwen/Qwen-7B4.28/4.30/4.34/4.36THUDM/chatglm3-6b4.34/4.36mistralai/Mistral-7B-v0.14.34/4.36ReferenceIf you find SignRound useful for your research, please cite our paper:@article{cheng2023optimize,title={OptimizeWeightRoundingviaSignedGradientDescentfortheQuantizationofLLMs},author={Cheng,WenhuaandZhang,WeiweiandShen,HaihaoandCai,YiyangandHe,XinandLv,Kaokao},journal={arXivpreprintarXiv:2309.05516},year={2023}}