V0507HMA15HV4

This model is a fine-tuned version of microsoft/phi-2 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -90.3589

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
-10.2471	0.09	10	-11.5048
-12.7013	0.18	20	-14.6171
-17.062	0.27	30	-21.1957
-25.2942	0.36	40	-31.1893
-35.8779	0.45	50	-42.0890
-45.8598	0.54	60	-51.0205
-54.2561	0.63	70	-59.2111
-62.3661	0.73	80	-66.9216
-69.4503	0.82	90	-73.1015
-75.6331	0.91	100	-79.7275
-80.4469	1.0	110	-79.1007
-82.9062	1.09	120	-78.7341
-79.1302	1.18	130	-78.9765
-80.7856	1.27	140	-81.2231
-82.8251	1.36	150	-84.6507
-85.8154	1.45	160	-89.2273
-87.8705	1.54	170	-86.6222
-87.8482	1.63	180	-87.4917
-89.9952	1.72	190	-85.7639
-88.213	1.81	200	-88.9700
-90.884	1.9	210	-89.7730
-90.422	1.99	220	-89.8904
-89.8791	2.08	230	-90.1639
-90.1325	2.18	240	-90.1416
-90.6013	2.27	250	-89.7608
-89.5485	2.36	260	-89.8643
-90.4119	2.45	270	-90.7045
-90.9852	2.54	280	-90.5243
-90.4856	2.63	290	-90.2753
-90.5832	2.72	300	-90.5374
-90.7252	2.81	310	-90.4361
-90.5843	2.9	320	-90.3612
-90.4331	2.99	330	-90.3589

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.18.0
• Tokenizers 0.14.1