Hugging Face's logo

ale-bay
/
zephyr-7b-dpo-qlora

PEFT
TensorBoard
Safetensors
mistral
alignment-handbook
trl
dpo
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
ale-bay commited on
Commit
ec88161
•
1 Parent(s): a10c986

Model save

Browse files
Files changed (5) hide show
  1. README.md +32 -38
  2. adapter_model.safetensors +1 -1
  3. all_results.json +5 -18
  4. train_results.json +5 -5
  5. trainer_state.json +0 -0
README.md CHANGED
@@ -2,16 +2,10 @@
2
  license: apache-2.0
3
  library_name: peft
4
  tags:
5
- - alignment-handbook
6
- - trl
7
- - dpo
8
- - generated_from_trainer
9
  - trl
10
  - dpo
11
  - generated_from_trainer
12
  base_model: mistralai/Mistral-7B-v0.1
13
- datasets:
14
- - HuggingFaceH4/ultrafeedback_binarized
15
  model-index:
16
  - name: zephyr-7b-dpo-qlora
17
  results: []
@@ -22,17 +16,17 @@ should probably proofread and complete it, then remove this comment. -->
22
 
23
  # zephyr-7b-dpo-qlora
24
 
25
- This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-qlora](https://huggingface.co/alignment-handbook/zephyr-7b-sft-qlora) on the HuggingFaceH4/ultrafeedback_binarized dataset.
26
  It achieves the following results on the evaluation set:
27
- - Loss: 0.4920
28
- - Rewards/chosen: -2.5098
29
- - Rewards/rejected: -3.5905
30
- - Rewards/accuracies: 0.7560
31
- - Rewards/margins: 1.0807
32
- - Logps/rejected: -600.3103
33
- - Logps/chosen: -516.2818
34
- - Logits/rejected: 2.5098
35
- - Logits/chosen: 2.2972
36
 
37
  ## Model description
38
 
@@ -69,31 +63,31 @@ The following hyperparameters were used during training:
69
 
70
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
71
  |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
72
- | 0.6622 | 0.05 | 100 | 0.6637 | 0.0126 | -0.0636 | 0.6840 | 0.0762 | -247.6176 | -264.0424 | -2.2973 | -2.3242 |
73
- | 0.6069 | 0.1 | 200 | 0.6175 | -0.5399 | -0.8086 | 0.6720 | 0.2687 | -322.1209 | -319.2918 | -1.9985 | -2.0644 |
74
- | 0.5858 | 0.16 | 300 | 0.5707 | -0.8385 | -1.3622 | 0.6930 | 0.5238 | -377.4863 | -349.1537 | 0.2196 | 0.1195 |
75
- | 0.5518 | 0.21 | 400 | 0.5536 | -0.8070 | -1.4119 | 0.7230 | 0.6049 | -382.4471 | -346.0015 | 0.8423 | 0.7208 |
76
- | 0.5953 | 0.26 | 500 | 0.5575 | -0.6678 | -1.1831 | 0.7110 | 0.5153 | -359.5695 | -332.0846 | 1.2558 | 1.0708 |
77
- | 0.5032 | 0.31 | 600 | 0.5359 | -1.3551 | -2.1333 | 0.7310 | 0.7782 | -454.5939 | -400.8145 | 2.8427 | 2.7062 |
78
- | 0.5741 | 0.37 | 700 | 0.5317 | -1.2904 | -2.0407 | 0.7260 | 0.7503 | -445.3269 | -394.3451 | 3.1371 | 2.9904 |
79
- | 0.5318 | 0.42 | 800 | 0.5149 | -1.6058 | -2.4688 | 0.7450 | 0.8630 | -488.1442 | -425.8877 | 3.7140 | 3.5383 |
80
- | 0.5353 | 0.47 | 900 | 0.5125 | -2.5710 | -3.5411 | 0.7460 | 0.9701 | -595.3752 | -522.4096 | 4.4179 | 4.2065 |
81
- | 0.574 | 0.52 | 1000 | 0.5035 | -2.6228 | -3.6684 | 0.7370 | 1.0456 | -608.1039 | -527.5898 | 2.6517 | 2.4408 |
82
- | 0.471 | 0.58 | 1100 | 0.5028 | -2.6309 | -3.7142 | 0.75 | 1.0833 | -612.6806 | -528.3990 | 2.2637 | 2.0694 |
83
- | 0.4888 | 0.63 | 1200 | 0.4965 | -2.4412 | -3.4135 | 0.7530 | 0.9723 | -582.6143 | -509.4261 | 2.4042 | 2.2263 |
84
- | 0.5204 | 0.68 | 1300 | 0.4941 | -2.2701 | -3.2940 | 0.7480 | 1.0239 | -570.6591 | -492.3148 | 2.2065 | 2.0121 |
85
- | 0.5158 | 0.73 | 1400 | 0.4925 | -2.6194 | -3.7070 | 0.7540 | 1.0875 | -611.9571 | -527.2493 | 2.4817 | 2.2784 |
86
- | 0.4677 | 0.79 | 1500 | 0.4922 | -2.6220 | -3.7128 | 0.7540 | 1.0908 | -612.5421 | -527.5074 | 2.5848 | 2.3739 |
87
- | 0.5464 | 0.84 | 1600 | 0.4925 | -2.5137 | -3.5972 | 0.7510 | 1.0835 | -600.9805 | -516.6763 | 2.4955 | 2.2803 |
88
- | 0.5078 | 0.89 | 1700 | 0.4920 | -2.5031 | -3.5840 | 0.7550 | 1.0809 | -599.6627 | -515.6122 | 2.5160 | 2.3031 |
89
- | 0.4864 | 0.94 | 1800 | 0.4921 | -2.5103 | -3.5902 | 0.7550 | 1.0799 | -600.2827 | -516.3320 | 2.5115 | 2.2982 |
90
- | 0.5211 | 0.99 | 1900 | 0.4921 | -2.5098 | -3.5900 | 0.7550 | 1.0803 | -600.2638 | -516.2831 | 2.5098 | 2.2971 |
91
 
92
 
93
  ### Framework versions
94
 
95
  - PEFT 0.7.1
96
- - Transformers 4.38.2
97
- - Pytorch 2.1.2
98
- - Datasets 2.14.6
99
  - Tokenizers 0.15.2
 
2
  license: apache-2.0
3
  library_name: peft
4
  tags:
 
 
 
 
5
  - trl
6
  - dpo
7
  - generated_from_trainer
8
  base_model: mistralai/Mistral-7B-v0.1
 
 
9
  model-index:
10
  - name: zephyr-7b-dpo-qlora
11
  results: []
 
16
 
17
  # zephyr-7b-dpo-qlora
18
 
19
+ This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on an unknown dataset.
20
  It achieves the following results on the evaluation set:
21
+ - Loss: 0.4974
22
+ - Rewards/chosen: -2.4531
23
+ - Rewards/rejected: -3.4740
24
+ - Rewards/accuracies: 0.75
25
+ - Rewards/margins: 1.0209
26
+ - Logps/rejected: -595.1221
27
+ - Logps/chosen: -517.0148
28
+ - Logits/rejected: -1.3432
29
+ - Logits/chosen: -1.4359
30
 
31
  ## Model description
32
 
 
63
 
64
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
65
  |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
66
+ | 0.6641 | 0.05 | 100 | 0.6636 | 0.0054 | -0.0681 | 0.6900 | 0.0735 | -254.5337 | -271.1659 | -2.0436 | -2.1368 |
67
+ | 0.6105 | 0.1 | 200 | 0.6075 | -0.3236 | -0.5938 | 0.6890 | 0.2702 | -307.0967 | -304.0613 | -2.0030 | -2.0919 |
68
+ | 0.5883 | 0.16 | 300 | 0.5817 | -0.7122 | -1.1286 | 0.7020 | 0.4164 | -360.5768 | -342.9188 | -1.9914 | -2.0761 |
69
+ | 0.5651 | 0.21 | 400 | 0.5665 | -0.7901 | -1.2897 | 0.7250 | 0.4996 | -376.6874 | -350.7093 | -1.9001 | -1.9820 |
70
+ | 0.5136 | 0.26 | 500 | 0.5520 | -1.0330 | -1.6646 | 0.7190 | 0.6316 | -414.1808 | -374.9992 | -1.8081 | -1.8880 |
71
+ | 0.5587 | 0.31 | 600 | 0.5327 | -1.3215 | -2.0089 | 0.7320 | 0.6874 | -448.6079 | -403.8534 | -1.4665 | -1.5609 |
72
+ | 0.5167 | 0.37 | 700 | 0.5299 | -1.2797 | -2.1992 | 0.7230 | 0.9196 | -467.6413 | -399.6684 | -1.3918 | -1.4903 |
73
+ | 0.5465 | 0.42 | 800 | 0.5189 | -1.6646 | -2.4686 | 0.7200 | 0.8041 | -494.5844 | -438.1617 | -1.3685 | -1.4642 |
74
+ | 0.5002 | 0.47 | 900 | 0.5142 | -1.7844 | -2.7217 | 0.7290 | 0.9373 | -519.8885 | -450.1383 | -1.4179 | -1.5054 |
75
+ | 0.5017 | 0.52 | 1000 | 0.5058 | -2.6175 | -3.6120 | 0.7360 | 0.9946 | -608.9218 | -533.4493 | -1.2973 | -1.3948 |
76
+ | 0.4966 | 0.58 | 1100 | 0.5043 | -2.0581 | -2.9819 | 0.7370 | 0.9239 | -545.9103 | -477.5080 | -1.3783 | -1.4740 |
77
+ | 0.5087 | 0.63 | 1200 | 0.5040 | -2.3715 | -3.3475 | 0.7450 | 0.9760 | -582.4712 | -508.8495 | -1.3331 | -1.4262 |
78
+ | 0.4799 | 0.68 | 1300 | 0.5011 | -2.3067 | -3.3444 | 0.7450 | 1.0377 | -582.1562 | -502.3687 | -1.3340 | -1.4277 |
79
+ | 0.4606 | 0.73 | 1400 | 0.4991 | -2.5016 | -3.5583 | 0.7430 | 1.0567 | -603.5469 | -521.8631 | -1.3291 | -1.4219 |
80
+ | 0.4763 | 0.79 | 1500 | 0.4985 | -2.4979 | -3.5204 | 0.7470 | 1.0225 | -599.7631 | -521.4944 | -1.3394 | -1.4325 |
81
+ | 0.5008 | 0.84 | 1600 | 0.4977 | -2.4555 | -3.4719 | 0.7480 | 1.0164 | -594.9102 | -517.2504 | -1.3492 | -1.4415 |
82
+ | 0.4654 | 0.89 | 1700 | 0.4976 | -2.4498 | -3.4672 | 0.7510 | 1.0174 | -594.4417 | -516.6852 | -1.3478 | -1.4402 |
83
+ | 0.4854 | 0.94 | 1800 | 0.4975 | -2.4526 | -3.4731 | 0.7480 | 1.0205 | -595.0339 | -516.9640 | -1.3441 | -1.4366 |
84
+ | 0.4879 | 0.99 | 1900 | 0.4974 | -2.4531 | -3.4740 | 0.75 | 1.0209 | -595.1221 | -517.0148 | -1.3432 | -1.4359 |
85
 
86
 
87
  ### Framework versions
88
 
89
  - PEFT 0.7.1
90
+ - Transformers 4.39.3
91
+ - Pytorch 2.3.0+cu121
92
+ - Datasets 2.19.1
93
  - Tokenizers 0.15.2
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1d17e769ba48d58042fa7f0a40e63409b72e0d70d0d20ff853ea765201a29121
3
  size 671150064
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7b408f44a61689a8466c178933f9b3f575cfed42b20ffb147556750c13f45448
3
  size 671150064
all_results.json CHANGED
@@ -1,21 +1,8 @@
1
  {
2
  "epoch": 1.0,
3
- "eval_logits/chosen": 2.2971765995025635,
4
- "eval_logits/rejected": 2.509765625,
5
- "eval_logps/chosen": -516.2817993164062,
6
- "eval_logps/rejected": -600.310302734375,
7
- "eval_loss": 0.4919675290584564,
8
- "eval_rewards/accuracies": 0.7559999823570251,
9
- "eval_rewards/chosen": -2.5097556114196777,
10
- "eval_rewards/margins": 1.0807288885116577,
11
- "eval_rewards/rejected": -3.590484380722046,
12
- "eval_runtime": 449.3054,
13
- "eval_samples": 2000,
14
- "eval_samples_per_second": 4.451,
15
- "eval_steps_per_second": 0.278,
16
- "train_loss": 0.528951391499704,
17
- "train_runtime": 37787.7404,
18
- "train_samples": 61135,
19
- "train_samples_per_second": 1.618,
20
- "train_steps_per_second": 0.051
21
  }
 
1
  {
2
  "epoch": 1.0,
3
+ "train_loss": 0.5312882837824796,
4
+ "train_runtime": 19005.9968,
5
+ "train_samples": 61134,
6
+ "train_samples_per_second": 3.217,
7
+ "train_steps_per_second": 0.1
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  }
train_results.json CHANGED
@@ -1,8 +1,8 @@
1
  {
2
  "epoch": 1.0,
3
- "train_loss": 0.528951391499704,
4
- "train_runtime": 37787.7404,
5
- "train_samples": 61135,
6
- "train_samples_per_second": 1.618,
7
- "train_steps_per_second": 0.051
8
  }
 
1
  {
2
  "epoch": 1.0,
3
+ "train_loss": 0.5312882837824796,
4
+ "train_runtime": 19005.9968,
5
+ "train_samples": 61134,
6
+ "train_samples_per_second": 3.217,
7
+ "train_steps_per_second": 0.1
8
  }
trainer_state.json CHANGED
The diff for this file is too large to render. See raw diff