V0507HMA15HV2

This model is a fine-tuned version of microsoft/phi-2 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -95.0498

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
-8.6532	0.09	10	-10.3327
-11.8451	0.18	20	-14.2014
-17.3537	0.27	30	-22.7794
-28.8388	0.36	40	-38.5512
-47.1885	0.45	50	-59.6364
-67.2573	0.54	60	-76.6591
-81.223	0.63	70	-86.3414
-87.9651	0.73	80	-90.0475
-91.3192	0.82	90	-92.4350
-92.7456	0.91	100	-93.1825
-93.4032	1.0	110	-93.7378
-93.8855	1.09	120	-93.9331
-94.0075	1.18	130	-93.9987
-94.001	1.27	140	-94.3115
-94.3566	1.36	150	-94.4505
-94.3346	1.45	160	-94.2625
-94.5793	1.54	170	-94.3309
-93.2701	1.63	180	-93.4388
-94.2829	1.72	190	-93.8681
-94.6778	1.81	200	-94.7489
-94.5762	1.9	210	-94.7745
-94.8427	1.99	220	-94.8903
-94.8653	2.08	230	-94.8499
-94.9237	2.18	240	-94.9720
-95.0027	2.27	250	-94.9841
-94.9404	2.36	260	-94.8479
-94.9594	2.45	270	-95.0076
-95.0772	2.54	280	-95.0798
-95.0775	2.63	290	-95.0480
-95.0528	2.72	300	-95.0415
-95.0652	2.81	310	-95.0442
-95.0738	2.9	320	-95.0494
-95.0694	2.99	330	-95.0498

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.18.0
• Tokenizers 0.14.1