---
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.1
tags:
- generated_from_trainer
model-index:
- name: Mistral-7B-v0.1-dpo-10k
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Mistral-7B-v0.1-dpo-10k

This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.7523
- Rewards/real: 2.2447
- Rewards/generated: 1.4806
- Rewards/accuracies: 0.6154
- Rewards/margins: 0.7641
- Logps/generated: -106.5099
- Logps/real: -116.4675
- Logits/generated: -2.3563
- Logits/real: -2.3976

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Rewards/real | Rewards/generated | Rewards/accuracies | Rewards/margins | Logps/generated | Logps/real | Logits/generated | Logits/real |
|:-------------:|:------:|:----:|:---------------:|:------------:|:-----------------:|:------------------:|:---------------:|:---------------:|:----------:|:----------------:|:-----------:|
| 0.74          | 0.1984 | 62   | 0.7414          | 1.1355       | 0.8829            | 0.6154             | 0.2526          | -112.4863       | -127.5589  | -2.4229          | -2.4711     |
| 0.7524        | 0.3968 | 124  | 0.7002          | 1.7305       | 1.2540            | 0.6923             | 0.4765          | -108.7756       | -121.6096  | -2.5561          | -2.5864     |
| 0.8028        | 0.5952 | 186  | 0.7025          | 1.7197       | 1.2525            | 0.6538             | 0.4673          | -108.7909       | -121.7167  | -2.4102          | -2.3984     |
| 0.7502        | 0.7936 | 248  | 0.7088          | 1.5388       | 0.9514            | 0.6346             | 0.5875          | -111.8017       | -123.5257  | -2.5032          | -2.5135     |
| 0.8621        | 0.992  | 310  | 0.7444          | 1.5171       | 1.1213            | 0.6731             | 0.3957          | -110.1023       | -123.7435  | -2.4965          | -2.5022     |
| 0.3246        | 1.1904 | 372  | 0.7215          | 2.3223       | 1.7036            | 0.6731             | 0.6187          | -104.2799       | -115.6916  | -2.5671          | -2.5848     |
| 0.3153        | 1.3888 | 434  | 0.7150          | 2.3474       | 1.7021            | 0.6538             | 0.6453          | -104.2945       | -115.4398  | -2.4999          | -2.5255     |
| 0.4053        | 1.5872 | 496  | 0.7083          | 2.2991       | 1.6619            | 0.6731             | 0.6372          | -104.6970       | -115.9233  | -2.4039          | -2.4069     |
| 0.3611        | 1.7856 | 558  | 0.7119          | 2.3331       | 1.7045            | 0.6731             | 0.6286          | -104.2702       | -115.5829  | -2.4323          | -2.4364     |
| 0.3933        | 1.984  | 620  | 0.7168          | 2.3292       | 1.7024            | 0.6731             | 0.6268          | -104.2917       | -115.6223  | -2.4321          | -2.4267     |
| 0.226         | 2.1824 | 682  | 0.7430          | 2.2194       | 1.4536            | 0.6346             | 0.7658          | -106.7797       | -116.7200  | -2.3994          | -2.4211     |
| 0.2117        | 2.3808 | 744  | 0.7449          | 2.1435       | 1.3976            | 0.5962             | 0.7459          | -107.3397       | -117.4795  | -2.4077          | -2.4527     |
| 0.2304        | 2.5792 | 806  | 0.7553          | 2.2242       | 1.4834            | 0.5769             | 0.7408          | -106.4812       | -116.6720  | -2.3411          | -2.3926     |
| 0.2423        | 2.7776 | 868  | 0.7526          | 2.2896       | 1.5597            | 0.5962             | 0.7299          | -105.7187       | -116.0179  | -2.3574          | -2.3974     |
| 0.2881        | 2.976  | 930  | 0.7523          | 2.2447       | 1.4806            | 0.6154             | 0.7641          | -106.5099       | -116.4675  | -2.3563          | -2.3976     |


### Framework versions

- Transformers 4.43.3
- Pytorch 2.2.2+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1