Model save

README.md

@@ -2,15 +2,9 @@
 license: apache-2.0
 base_model: alignment-handbook/zephyr-7b-sft-full
 tags:
-- alignment-handbook
 - trl
 - dpo
 - generated_from_trainer
-- trl
-- dpo
-- generated_from_trainer
-datasets:
-- HuggingFaceH4/ultrafeedback_binarized
 model-index:
 - name: zephyr-7b-dpo-full
   results: []
@@ -21,17 +15,17 @@ should probably proofread and complete it, then remove this comment. -->
 
 # zephyr-7b-dpo-full
 
-This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the HuggingFaceH4/ultrafeedback_binarized dataset.
+This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.0393
-- Rewards/chosen: -30.8792
-- Rewards/rejected: -34.9336
-- Rewards/accuracies: 1.0
-- Rewards/margins: 4.0543
-- Logps/rejected: -961.3383
-- Logps/chosen: -948.7985
-- Logits/rejected: -1.6730
-- Logits/chosen: -1.7277
+- Loss: 0.6824
+- Rewards/chosen: -4.2277
+- Rewards/rejected: -7.2864
+- Rewards/accuracies: 0.7773
+- Rewards/margins: 3.0587
+- Logps/rejected: -408.3961
+- Logps/chosen: -347.1476
+- Logits/rejected: -0.8310
+- Logits/chosen: -1.2135
 
 ## Model description
 
@@ -62,16 +56,26 @@ The following hyperparameters were used during training:
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_ratio: 0.1
-- num_epochs: 1
+- num_epochs: 3
 
 ### Training results
 
-| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
-|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
-| 0.2032        | 0.21  | 100  | 0.1514          | -18.9619       | -21.8043         | 0.9844             | 2.8424          | -698.7539      | -683.9697    | -0.9770         | -1.0944       |
-| 0.05          | 0.42  | 200  | 0.0465          | -27.8849       | -31.5859         | 1.0                | 3.7010          | -894.3857      | -882.2585    | -1.6112         | -1.6739       |
-| 0.0325        | 0.63  | 300  | 0.0437          | -31.3531       | -35.4469         | 0.9922             | 4.0938          | -971.6055      | -959.3290    | -1.4754         | -1.5406       |
-| 0.0337        | 0.84  | 400  | 0.0398          | -30.6169       | -34.6409         | 1.0                | 4.0239          | -955.4847      | -942.9697    | -1.7039         | -1.7576       |
+| Training Loss | Epoch | Step | Logits/chosen | Logits/rejected | Logps/chosen | Logps/rejected | Validation Loss | Rewards/accuracies | Rewards/chosen | Rewards/margins | Rewards/rejected |
+|:-------------:|:-----:|:----:|:-------------:|:---------------:|:------------:|:--------------:|:---------------:|:------------------:|:--------------:|:---------------:|:----------------:|
+| 0.582         | 0.21  | 100  | -2.5812       | -2.5431         | -254.2386    | -263.2876      | 0.5878          | 0.7188             | 0.4177         | 0.4488          | -0.0310          |
+| 0.558         | 0.42  | 200  | -2.3893       | -2.3398         | -261.4734    | -280.4191      | 0.5196          | 0.7773             | 0.0560         | 0.9436          | -0.8876          |
+| 0.4914        | 0.63  | 300  | -2.3653       | -2.3039         | -264.2936    | -286.6201      | 0.5110          | 0.7656             | -0.0850        | 1.1126          | -1.1976          |
+| 0.4922        | 0.84  | 400  | -2.3145       | -2.2570         | -263.2854    | -285.4248      | 0.5095          | 0.7852             | -0.0346        | 1.1033          | -1.1379          |
+| 0.1908        | 1.05  | 500  | -2.2442       | -2.1660         | -269.8426    | -300.6474      | 0.5179          | 0.7852             | -0.3625        | 1.5366          | -1.8990          |
+| 0.1675        | 1.26  | 600  | -2.2220       | -2.1249         | -287.2300    | -324.0812      | 0.5377          | 0.8008             | -1.2318        | 1.8389          | -3.0707          |
+| 0.1567        | 1.46  | 700  | -2.0453       | -1.9285         | -298.7820    | -333.3354      | 0.5348          | 0.7891             | -1.8094        | 1.7240          | -3.5334          |
+| 0.1475        | 1.67  | 800  | -2.2409       | -2.1202         | -296.3533    | -332.4951      | 0.5382          | 0.8008             | -1.6880        | 1.8034          | -3.4914          |
+| 0.1422        | 1.88  | 900  | -2.1980       | -2.0630         | -296.0324    | -335.6016      | 0.5518          | 0.7852             | -1.6719        | 1.9748          | -3.6467          |
+| 0.044         | 2.09  | 1000 | -1.7406       | -1.4629         | -316.4520    | -365.4959      | 0.6058          | 0.7891             | -2.6929        | 2.4485          | -5.1414          |
+| 0.0307        | 2.3   | 1100 | -1.3310       | -0.9162         | -337.0383    | -397.1617      | 0.6700          | 0.7695             | -3.7222        | 3.0025          | -6.7247          |
+| 0.0317        | 2.51  | 1200 | 0.6711        | -3.9616         | -6.9639      | 0.7773         | 3.0023          | -401.9448          | -341.8261      | -0.9227         | -1.2927          |
+| 0.0264        | 2.72  | 1300 | 0.6778        | -4.2314         | -7.2584      | 0.7773         | 3.0270          | -407.8352          | -347.2216      | -0.8370         | -1.2190          |
+| 0.0343        | 2.93  | 1400 | 0.6824        | -4.2277         | -7.2864      | 0.7773         | 3.0587          | -408.3961          | -347.1476      | -0.8310         | -1.2135          |
 
 
 ### Framework versions

all_results.json

@@ -1,21 +1,8 @@
 {
-    "epoch": 1.0,
-    "eval_logits/chosen": -1.7276861667633057,
-    "eval_logits/rejected": -1.6730353832244873,
-    "eval_logps/chosen": -948.7984619140625,
-    "eval_logps/rejected": -961.3382568359375,
-    "eval_loss": 0.03925349563360214,
-    "eval_rewards/accuracies": 1.0,
-    "eval_rewards/chosen": -30.879230499267578,
-    "eval_rewards/margins": 4.054325103759766,
-    "eval_rewards/rejected": -34.933555603027344,
-    "eval_runtime": 96.1529,
-    "eval_samples": 2000,
-    "eval_samples_per_second": 20.8,
-    "eval_steps_per_second": 0.333,
-    "train_loss": 0.14747725651603363,
-    "train_runtime": 7551.087,
+    "epoch": 3.0,
+    "train_loss": 0.007373018379906397,
+    "train_runtime": 5373.2533,
     "train_samples": 61134,
-    "train_samples_per_second": 8.096,
-    "train_steps_per_second": 0.063
+    "train_samples_per_second": 34.132,
+    "train_steps_per_second": 0.267
 }

config.json

@@ -20,7 +20,7 @@
   "sliding_window": 4096,
   "tie_word_embeddings": false,
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.40.1",
+  "transformers_version": "4.38.2",
   "use_cache": false,
   "vocab_size": 32000
 }

model-00001-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e072211b80a3d503c5e0f47e58bd5975f0d38179db79bda828a67533ba29b6cc
+oid sha256:f6108f92c0bc4c2834053762de98cc5c2078cf240bf3f4bf2226a02dc4314d19
 size 4943162336

model-00002-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fffe1f9d065e794a0381bcd43c1b3c930381febe5544c684c3c8fe3c3cbf4d84
+oid sha256:8dd45abf9705483eca1786a8485bfa9e8fce12425a0630df8190c9074c81ae2e
 size 4999819336

model-00003-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a1b5b4ce5ac68002586866a9d222ad03a2472fc41053e98ac2318f87b590e917
+oid sha256:9c43191053486250291b313a6e33349fc47cb611fa7ee1af51bdfba716087d7e
 size 4540516344

tokenizer.json

@@ -134,7 +134,6 @@
     "end_of_word_suffix": null,
     "fuse_unk": true,
     "byte_fallback": true,
-    "ignore_merges": false,
     "vocab": {
       "<unk>": 0,
       "<s>": 1,

train_results.json

@@ -1,8 +1,8 @@
 {
-    "epoch": 1.0,
-    "train_loss": 0.14747725651603363,
-    "train_runtime": 7551.087,
+    "epoch": 3.0,
+    "train_loss": 0.007373018379906397,
+    "train_runtime": 5373.2533,
     "train_samples": 61134,
-    "train_samples_per_second": 8.096,
-    "train_steps_per_second": 0.063
+    "train_samples_per_second": 34.132,
+    "train_steps_per_second": 0.267
 }

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fa634d33e851e8d288d42ce63c89f01941b5065c07c2a791cc14483791384143
-size 6328
+oid sha256:82aa70e8e41e01e95fd40baeec15296fe100e724322a1af2cfa9b1c3f6880ba4
+size 6264