metadata
datasets:
- CheshireAI/guanaco-unchained
Let's see how this goes.
Training in 8 bit and at full context. Is 8bit even a qlora?
python qlora.py \
--model_name_or_path /UI/text-generation-webui/models/llama-30b \
--output_dir ./output/guanaco-33b \
--logging_steps 1 \
--save_strategy steps \
--data_seed 42 \
--save_steps 69 \
--save_total_limit 999 \
--per_device_eval_batch_size 1 \
--dataloader_num_workers 3 \
--group_by_length \
--logging_strategy steps \
--remove_unused_columns False \
--do_train \
--do_eval false \
--do_mmlu_eval false \
--lora_r 64 \
--lora_alpha 16 \
--lora_modules all \
--bf16 \
--bits 8 \
--warmup_ratio 0.03 \
--lr_scheduler_type constant \
--gradient_checkpointing \
--gradient_accumulation_steps 32 \
--dataset oasst1 \
--source_max_len 2048 \
--target_max_len 2048 \
--per_device_train_batch_size 1 \
--num_train_epochs 3 \
--learning_rate 0.0001 \
--adam_beta2 0.999 \
--max_grad_norm 0.3 \
--lora_dropout 0.05 \
--weight_decay 0.0 \
--seed 0