zephyr-7b-dpo-full

This model was trained from scratch on the None dataset. It achieves the following results on the evaluation set:

• Logits/chosen: -0.9126
• Logits/rejected: -0.7601
• Logps/chosen: -361.4586
• Logps/rejected: -470.9522
• Loss: 0.4905
• Rewards/accuracies: 0.7539
• Rewards/chosen: -0.7983
• Rewards/margins: 1.0337
• Rewards/rejected: -1.8319

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Logits/chosen	Logits/rejected	Logps/chosen	Logps/rejected	Validation Loss	Rewards/accuracies	Rewards/chosen	Rewards/margins	Rewards/rejected
0.5759	0.21	100	-1.9151	-1.8988	-335.8161	-395.6910	0.5723	0.7148	-0.5418	0.5375	-1.0793
0.5391	0.42	200	-1.6570	-1.5774	-321.1682	-405.0056	0.5138	0.7461	-0.3953	0.7771	-1.1724
0.4788	0.63	300	-0.9864	-0.8826	-367.7277	-475.5532	0.4939	0.7578	-0.8609	1.0170	-1.8779
0.4937	0.84	400	-0.9126	-0.7601	-361.4586	-470.9522	0.4905	0.7539	-0.7983	1.0337	-1.8319

Framework versions

• Transformers 4.38.2
• Pytorch 2.1.2+cu118
• Datasets 2.16.1
• Tokenizers 0.15.2