|
--- |
|
license: mit |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
tags: |
|
- 'quantization ' |
|
- lora |
|
- loftq |
|
- llama |
|
--- |
|
# LoftQ Initialization |
|
|
|
| [Paper](https://arxiv.org/abs/2310.08659) | [Code](https://github.com/yxli2123/LoftQ) | [PEFT Example](https://github.com/huggingface/peft/tree/main/examples/loftq_finetuning) | |
|
|
|
LoftQ (LoRA-fine-tuning-aware Quantization) provides a quantized backbone Q and LoRA adapters A and B, given a full-precision pre-trained weight W. |
|
|
|
This model, `Meta-Llama-3-70B-4bit-64rank-1iter`, is obtained from [LLAMA-3-70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B). |
|
The backbone is under `LoftQ/Meta-Llama-3-70B-4bit-64rank-1iter` and LoRA adapters are under the `subfolder='loftq_init'`. |
|
|
|
## Model Info |
|
### Backbone |
|
- Size: ~ 43 GiB |
|
- Loaded format: bitsandbytes nf4 |
|
- Size loaded on GPU: ~43 GiB |
|
|
|
### LoRA adapters |
|
- rank: 64 |
|
- lora_alpha: 16 |
|
- target_modules: ["down_proj", "up_proj", "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"] |
|
|
|
## Usage |
|
|
|
**Training.** Here's an example of loading this model and preparing for the LoRA fine-tuning. |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModelForCausalLM, BitsAndBytesConfig |
|
from peft import PeftModel |
|
|
|
MODEL_ID = "LoftQ/Meta-Llama-3-70B-4bit-64rank-1iter" |
|
|
|
base_model = AutoModelForCausalLM.from_pretrained( |
|
MODEL_ID, |
|
torch_dtype=torch.bfloat16, # you may change it with different models |
|
quantization_config=BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_compute_dtype=torch.bfloat16, # bfloat16 is recommended |
|
bnb_4bit_use_double_quant=False, |
|
bnb_4bit_quant_type='nf4', |
|
), |
|
) |
|
peft_model = PeftModel.from_pretrained( |
|
base_model, |
|
MODEL_ID, |
|
subfolder="loftq_init", |
|
is_trainable=True, |
|
) |
|
|
|
# Do training with peft_model ... |
|
``` |
|
|
|
|
|
**Inference.** Here is an example code for inference after the model has been fine-tuned. |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModelForCausalLM, BitsAndBytesConfig |
|
from peft import PeftModel |
|
|
|
MODEL_ID = "LoftQ/Meta-Llama-3-70B-4bit-64rank-1iter" |
|
ADAPTER_PATH = "you/adapter/path" |
|
|
|
base_model = AutoModelForCausalLM.from_pretrained( |
|
MODEL_ID, |
|
torch_dtype=torch.bfloat16, # you may change it with different models |
|
quantization_config=BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_compute_dtype=torch.bfloat16, # bfloat16 is recommended |
|
bnb_4bit_use_double_quant=False, |
|
bnb_4bit_quant_type='nf4', |
|
), |
|
) |
|
peft_model = PeftModel.from_pretrained( |
|
base_model, |
|
ADAPTER_PATH, |
|
) |
|
|
|
# Do inference with peft_model ... |
|
``` |
|
|
|
See the full code at our [Github Repo]((https://github.com/yxli2123/LoftQ)) |
|
|
|
|
|
## Citation |
|
|
|
```bibtex |
|
@article{li2023loftq, |
|
title={Loftq: Lora-fine-tuning-aware quantization for large language models}, |
|
author={Li, Yixiao and Yu, Yifan and Liang, Chen and He, Pengcheng and Karampatziakis, Nikos and Chen, Weizhu and Zhao, Tuo}, |
|
journal={arXiv preprint arXiv:2310.08659}, |
|
year={2023} |
|
} |
|
``` |