phi-3-mini-LoRA

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.8445

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
1.0374	0.0329	100	1.0194
0.9715	0.0658	200	0.9314
0.9106	0.0987	300	0.8969
0.888	0.1316	400	0.8869
0.8902	0.1645	500	0.8813
0.8826	0.1974	600	0.8777
0.8763	0.2303	700	0.8745
0.8728	0.2632	800	0.8723
0.8707	0.2961	900	0.8701
0.8702	0.3289	1000	0.8684
0.8631	0.3618	1100	0.8664
0.8623	0.3947	1200	0.8647
0.8655	0.4276	1300	0.8624
0.863	0.4605	1400	0.8602
0.858	0.4934	1500	0.8586
0.859	0.5263	1600	0.8578
0.8527	0.5592	1700	0.8569
0.8587	0.5921	1800	0.8563
0.8551	0.625	1900	0.8557
0.8548	0.6579	2000	0.8550
0.8515	0.6908	2100	0.8546
0.8531	0.7237	2200	0.8542
0.8567	0.7566	2300	0.8535
0.8589	0.7895	2400	0.8532
0.8547	0.8224	2500	0.8529
0.8537	0.8553	2600	0.8525
0.85	0.8882	2700	0.8521
0.8518	0.9211	2800	0.8519
0.8456	0.9539	2900	0.8515
0.8585	0.9868	3000	0.8512
0.849	1.0197	3100	0.8509
0.8549	1.0526	3200	0.8507
0.8502	1.0855	3300	0.8504
0.8504	1.1184	3400	0.8502
0.8488	1.1513	3500	0.8500
0.8504	1.1842	3600	0.8497
0.8465	1.2171	3700	0.8495
0.8471	1.25	3800	0.8494
0.8467	1.2829	3900	0.8491
0.8439	1.3158	4000	0.8489
0.8467	1.3487	4100	0.8487
0.8461	1.3816	4200	0.8485
0.8525	1.4145	4300	0.8483
0.8426	1.4474	4400	0.8481
0.8479	1.4803	4500	0.8480
0.853	1.5132	4600	0.8478
0.8432	1.5461	4700	0.8477
0.8416	1.5789	4800	0.8475
0.8527	1.6118	4900	0.8474
0.849	1.6447	5000	0.8472
0.8446	1.6776	5100	0.8471
0.8427	1.7105	5200	0.8469
0.8464	1.7434	5300	0.8468
0.8444	1.7763	5400	0.8466
0.8479	1.8092	5500	0.8465
0.8452	1.8421	5600	0.8465
0.8387	1.875	5700	0.8466
0.845	1.9079	5800	0.8463
0.8402	1.9408	5900	0.8461
0.8459	1.9737	6000	0.8460
0.8431	2.0066	6100	0.8460
0.8395	2.0395	6200	0.8459
0.8395	2.0724	6300	0.8458
0.8457	2.1053	6400	0.8457
0.8438	2.1382	6500	0.8457
0.8411	2.1711	6600	0.8456
0.8386	2.2039	6700	0.8456
0.8393	2.2368	6800	0.8454
0.8406	2.2697	6900	0.8454
0.8386	2.3026	7000	0.8453
0.8456	2.3355	7100	0.8453
0.8424	2.3684	7200	0.8452
0.8437	2.4013	7300	0.8451
0.8426	2.4342	7400	0.8451
0.8393	2.4671	7500	0.8450
0.8398	2.5	7600	0.8450
0.8434	2.5329	7700	0.8449
0.8456	2.5658	7800	0.8449
0.8393	2.5987	7900	0.8449
0.8401	2.6316	8000	0.8448
0.838	2.6645	8100	0.8448
0.8432	2.6974	8200	0.8447
0.8471	2.7303	8300	0.8447
0.8435	2.7632	8400	0.8446
0.8441	2.7961	8500	0.8446
0.8399	2.8289	8600	0.8446
0.8391	2.8618	8700	0.8446
0.8432	2.8947	8800	0.8446
0.8459	2.9276	8900	0.8446
0.8446	2.9605	9000	0.8445
0.8412	2.9934	9100	0.8445

Framework versions

• PEFT 0.12.0
• Transformers 4.43.3
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1