training script
#1
by
tamdiep106
- opened
Hi
@KBlueLeaf
, thank you for this great implement and the pretrained model.
I tried to re-create your finetuning on my own dataset, i tried to used the finetune script in https://github.com/KohakuBlueleaf/guanaco-lora/blob/main/finetune.py but have not got any good result
Could you please provide the script along with parameter you use for finetune this model?
Thank you
Sorry for late response
@tamdiep106
Here is the command what I used to run this model (leh-v2
guanaco-lora\finetune.py --base_model decapoda-research/llama-7b-hf --data_path .\datasets\*.json --output_dir guanaco-leh-v2 --batch_size 128 --micro_batch_size 16 --num_epochs 2 --learning_rate 2e-4 --cutoff_len 1024 --val_set_size 0 --group_by_length True --train_on_inputs False --lora_r 16 --lora_alpha 16 --lora_dropout 0.1
If you have any question about the hyperparameters, just ask me!
Thank you so much @KBlueLeaf , this is all i need. Appreciate your support
tamdiep106
changed discussion status to
closed