Fine-tuning
- Base Model: NousResearch/Llama-2-7b-hf
- Dataset for fine-tuning: vicgalle/alpaca-gpt4
- Training
- BitsAndBytesConfig
BitsAndBytesConfig( load_in_4bit= True, bnb_4bit_quant_type= "nf4", bnb_4bit_compute_dtype= torch.bfloat16, bnb_4bit_use_double_quant= False, )
- LoRA Config
LoraConfig( r=16, lora_alpha= 8, # alpha = rank * 2 ! lora_dropout= 0.1, bias="none", task_type="CAUSAL_LM", target_modules=["q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj"] )
- Training Arguments
TrainingArguments( output_dir= "./results", num_train_epochs= 1, per_device_train_batch_size= 8, gradient_accumulation_steps= 2, optim = "paged_adamw_8bit", save_steps= 1000, logging_steps= 30, learning_rate= 2e-4, weight_decay= 0.001, fp16= False, bf16= False, max_grad_norm= 0.3, max_steps= -1, warmup_ratio= 0.3, group_by_length= True, lr_scheduler_type= "linear", report_to="wandb", )
- BitsAndBytesConfig
- Downloads last month
- 12
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.