openhermes-mistral-dpo-gptq

This model is a fine-tuned version of TheBloke/OpenHermes-2-Mistral-7B-GPTQ on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0323
• Rewards/chosen: -1.0831
• Rewards/rejected: -9.5400
• Rewards/accuracies: 1.0
• Rewards/margins: 8.4569
• Logps/rejected: -319.0484
• Logps/chosen: -82.1537
• Logits/rejected: -2.6023
• Logits/chosen: -2.2419

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2
• training_steps: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.556	0.0	10	0.4270	0.0181	-0.6933	1.0	0.7114	-230.5818	-71.1419	-2.6557	-2.3264
0.3645	0.0	20	0.1583	0.1759	-2.5866	1.0	2.7625	-249.5149	-69.5636	-2.6400	-2.3441
0.2037	0.0	30	0.0681	0.0985	-5.0474	1.0	5.1459	-274.1230	-70.3379	-2.6245	-2.3228
0.0315	0.0	40	0.0431	-0.9146	-8.5841	1.0	7.6695	-309.4896	-80.4684	-2.6094	-2.3791
0.0655	0.0	50	0.0323	-1.0831	-9.5400	1.0	8.4569	-319.0484	-82.1537	-2.6023	-2.2419

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.0.1+cu117
• Datasets 2.18.0
• Tokenizers 0.15.2