File size: 1,410 Bytes
a852d0a 84d7088 b0234a8 7722625 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
---
license: llama2
datasets:
- vicgalle/alpaca-gpt4
pipeline_tag: text-generation
language:
- en
tags:
- llama-2
---
## Fine-tuning
- Base Model: [NousResearch/Llama-2-7b-hf](https://huggingface.co/NousResearch/Llama-2-7b-hf)
- Dataset for fine-tuning: [vicgalle/alpaca-gpt4](https://huggingface.co/vicgalle/gpt2-alpaca-gpt4)
- Training
- BitsAndBytesConfig
```
BitsAndBytesConfig(
load_in_4bit= True,
bnb_4bit_quant_type= "nf4",
bnb_4bit_compute_dtype= torch.bfloat16,
bnb_4bit_use_double_quant= False,
)
```
- LoRA Config
```
LoraConfig(
r=16,
lora_alpha= 8, # alpha = rank * 2 !
lora_dropout= 0.1,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj"]
)
```
- Training Arguments
```
TrainingArguments(
output_dir= "./results",
num_train_epochs= 1,
per_device_train_batch_size= 8,
gradient_accumulation_steps= 2,
optim = "paged_adamw_8bit",
save_steps= 1000,
logging_steps= 30,
learning_rate= 2e-4,
weight_decay= 0.001,
fp16= False,
bf16= False,
max_grad_norm= 0.3,
max_steps= -1,
warmup_ratio= 0.3,
group_by_length= True,
lr_scheduler_type= "linear",
report_to="wandb",
)
``` |