phi-2-gpo-renew2-b0.001-extra-v2-i1

This model is a fine-tuned version of DUAL-GPO/phi-2-gpo-renew2-b0.001-i0 on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

• Loss: 0.0388
• Rewards/chosen: 0.0266
• Rewards/rejected: -0.0126
• Rewards/accuracies: 0.6070
• Rewards/margins: 0.0392
• Logps/rejected: -379.8497
• Logps/chosen: -369.7509
• Logits/rejected: -0.9196
• Logits/chosen: -0.9539

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.098	0.06	100	0.0533	-0.0029	-0.0036	0.4980	0.0007	-370.8433	-399.2503	-0.7225	-0.8171
0.094	0.13	200	0.0491	-0.0390	-0.0525	0.5525	0.0135	-419.6949	-435.2693	-1.0754	-1.1388
0.0898	0.19	300	0.0452	-0.0184	-0.0403	0.5780	0.0218	-407.5088	-414.7480	-1.0291	-1.0858
0.0731	0.26	400	0.0430	-0.0069	-0.0331	0.5970	0.0262	-400.2979	-403.1916	-0.9864	-1.0412
0.0787	0.32	500	0.0422	-0.0122	-0.0473	0.6070	0.0351	-414.4887	-408.4566	-1.0587	-1.0975
0.0742	0.38	600	0.0406	0.0135	-0.0175	0.6085	0.0309	-384.7105	-382.8363	-0.9872	-1.0246
0.0635	0.45	700	0.0401	0.0166	-0.0188	0.6095	0.0354	-386.0258	-379.6696	-0.9903	-1.0225
0.0881	0.51	800	0.0395	0.0250	-0.0102	0.6085	0.0352	-377.4323	-371.2672	-0.9658	-0.9975
0.0753	0.58	900	0.0393	0.0304	-0.0046	0.5990	0.0350	-371.7872	-365.8699	-0.9026	-0.9456
0.0922	0.64	1000	0.0390	0.0286	-0.0075	0.5990	0.0361	-374.7669	-367.7319	-0.8801	-0.9184
0.0703	0.7	1100	0.0389	0.0227	-0.0161	0.6000	0.0387	-383.3026	-373.6226	-0.9300	-0.9602
0.0746	0.77	1200	0.0388	0.0226	-0.0179	0.6050	0.0405	-385.1601	-373.7153	-0.8944	-0.9306
0.0925	0.83	1300	0.0387	0.0263	-0.0131	0.6030	0.0393	-380.3072	-370.0340	-0.9171	-0.9494
0.0863	0.9	1400	0.0387	0.0269	-0.0123	0.6055	0.0392	-379.5608	-369.4450	-0.9121	-0.9447
0.0904	0.96	1500	0.0386	0.0268	-0.0124	0.6045	0.0392	-379.6000	-369.4944	-0.9203	-0.9536

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.1.2
• Datasets 2.14.6
• Tokenizers 0.15.2