File size: 2,899 Bytes
96a8e58 2e3505c 7aa76f9 96a8e58 2e3505c 4323f60 2e3505c 4323f60 2e3505c 3b565ee 2e3505c 3b565ee 2e3505c 8a8a74e 2e3505c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
---
license: other
language:
- en
pipeline_tag: text2text-generation
tags:
- alpaca
- llama
- chat
- gpt4
inference: false
---
This is a 4bit 128g GPTQ of [chansung's gpt4-alpaca-lora-13b](https://huggingface.co/chansung/gpt4-alpaca-lora-13b).
## How to easily download and use this model in text-generation-webui
Open the text-generation-webui UI as normal.
1. Click the **Model tab**.
2. Under **Download custom model or LoRA**, enter `TheBloke/gpt4-alpaca-lora-13B-GPTQ-4bit-128g`.
3. Click **Download**.
4. Wait until it says it's finished downloading.
5. Click the **Refresh** icon next to **Model** in the top left.
6. In the **Model drop-down**: choose the model you just downloaded,`gpt4-alpaca-lora-13B-GPTQ-4bit-128g`.
7. If you see an error in the bottom right, ignore it - it's temporary.
8. Check that the `GPTQ parameters` are correct on the right: `Bits = 4`, `Groupsize = 128`, `model_type = Llama`
9. Click **Save settings for this model** in the top right.
10. Click **Reload the Model** in the top right.
11. Once it says it's loaded, click the **Text Generation tab** and enter a prompt!
Command to create was:
```
CUDA_VISIBLE_DEVICES=0 python3 llama.py /content/gpt4-alpaca-lora-13B-HF c4 --wbits 4 --true-sequential --act-order --groupsize 128 --save_safetensors /content/gpt4-alpaca-lora-13B-GPTQ-4bit-128g.safetensors
```
Command to clone the latest Triton GPTQ-for-LLaMa repo for inference using `llama_inference.py`, or in `text-generation-webui`:
```
# Clone text-generation-webui, if you don't already have it
git clone https://github.com/oobabooga/text-generation-webui
# Make a repositories directory
mkdir -p text-generation-webui/repositories
cd text-generation-webui/repositories
# Clone the latest GPTQ-for-LLaMa code inside text-generation-webui
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa
```
There is also a `no-act-order.safetensors` file which will work with oobabooga's fork of GPTQ-for-LLaMa; it does not require the latest GPTQ code.
# Original model card is below
This repository comes with LoRA checkpoint to make LLaMA into a chatbot like language model. The checkpoint is the output of instruction following fine-tuning process with the following settings on 8xA100(40G) DGX system.
- Training script: borrowed from the official [Alpaca-LoRA](https://github.com/tloen/alpaca-lora) implementation
- Training script:
```shell
python finetune.py \
--base_model='decapoda-research/llama-30b-hf' \
--data_path='alpaca_data_gpt4.json' \
--num_epochs=10 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./gpt4-alpaca-lora-30b' \
--lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
--lora_r=16 \
--batch_size=... \
--micro_batch_size=...
```
You can find how the training went from W&B report [here](https://wandb.ai/chansung18/gpt4_alpaca_lora/runs/w3syd157?workspace=user-chansung18).
|