Model save

Browse files

Files changed (11) hide show

README.md +19 -24
all_results.json +5 -4
config.json +2 -2
generation_config.json +1 -1
model-00001-of-00003.safetensors +1 -1
model-00002-of-00003.safetensors +1 -1
model-00003-of-00003.safetensors +1 -1
runs/May28_00-07-25_n136-112-146/events.out.tfevents.1716826553.n136-112-146.284037.0 +2 -2
train_results.json +5 -4
trainer_state.json +521 -1306
training_args.bin +2 -2

README.md CHANGED Viewed

@@ -13,17 +13,17 @@ should probably proofread and complete it, then remove this comment. -->
 # zephyr-7b-dpo-full
-This model was trained from scratch on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.3310
-- Rewards/chosen: -0.6343
-- Rewards/rejected: -2.0702
-- Rewards/accuracies: 0.8711
-- Rewards/margins: 1.4359
-- Logps/rejected: -579.6317
-- Logps/chosen: -442.8079
-- Logits/rejected: -4.3667
-- Logits/chosen: -3.9322
 ## Model description
@@ -42,7 +42,7 @@ More information needed
 ### Training hyperparameters
 The following hyperparameters were used during training:
-- learning_rate: 5e-07
 - train_batch_size: 8
 - eval_batch_size: 8
 - seed: 42
@@ -58,21 +58,16 @@ The following hyperparameters were used during training:
 ### Training results
-| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
-|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
-| 0.5557        | 0.12  | 100  | 0.6118          | -0.3644        | -0.7265          | 0.6719             | 0.3621          | -445.2614      | -415.8167    | -2.1569         | -2.1594       |
-| 0.4689        | 0.23  | 200  | 0.5068          | -0.4921        | -1.3016          | 0.75               | 0.8095          | -502.7699      | -428.5852    | -2.9279         | -2.8572       |
-| 0.4351        | 0.35  | 300  | 0.4574          | -0.5263        | -1.5234          | 0.7930             | 0.9971          | -524.9551      | -432.0108    | -3.6524         | -3.4654       |
-| 0.3978        | 0.46  | 400  | 0.4130          | -0.5219        | -1.7269          | 0.8359             | 1.2050          | -545.3044      | -431.5721    | -3.8428         | -3.5190       |
-| 0.422         | 0.58  | 500  | 0.3804          | -0.5284        | -1.7684          | 0.8516             | 1.2400          | -549.4502      | -432.2204    | -3.9749         | -3.6652       |
-| 0.3728        | 0.69  | 600  | 0.3498          | -0.6801        | -2.0888          | 0.8555             | 1.4087          | -581.4929      | -447.3842    | -4.3492         | -3.9204       |
-| 0.4072        | 0.81  | 700  | 0.3413          | -0.5876        | -1.9622          | 0.8711             | 1.3746          | -568.8267      | -438.1348    | -4.2357         | -3.8217       |
-| 0.388         | 0.92  | 800  | 0.3310          | -0.6343        | -2.0702          | 0.8711             | 1.4359          | -579.6317      | -442.8079    | -4.3667         | -3.9322       |
 ### Framework versions
-- Transformers 4.38.2
 - Pytorch 2.1.2+cu118
-- Datasets 2.16.1
-- Tokenizers 0.15.2

 # zephyr-7b-dpo-full
+This model was trained from scratch on an unknown dataset.
 It achieves the following results on the evaluation set:
+- Loss: 0.7111
+- Rewards/chosen: -0.8296
+- Rewards/rejected: -0.9542
+- Rewards/accuracies: 0.5625
+- Rewards/margins: 0.1246
+- Logps/rejected: -613.8080
+- Logps/chosen: -473.4393
+- Logits/rejected: -5.2824
+- Logits/chosen: -5.0285
 ## Model description
 ### Training hyperparameters
 The following hyperparameters were used during training:
+- learning_rate: 5e-08
 - train_batch_size: 8
 - eval_batch_size: 8
 - seed: 42
 ### Training results
+| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.5044        | 0.2558 | 100  | 0.7105          | -0.1129        | -0.0877          | 0.4727             | -0.0252         | -527.1570      | -401.7669    | -4.8307         | -4.6554       |
+| 0.3343        | 0.5115 | 200  | 0.6982          | -0.5200        | -0.6117          | 0.5586             | 0.0918          | -579.5609      | -442.4707    | -5.1101         | -4.8657       |
+| 0.2972        | 0.7673 | 300  | 0.7111          | -0.8296        | -0.9542          | 0.5625             | 0.1246          | -613.8080      | -473.4393    | -5.2824         | -5.0285       |
 ### Framework versions
+- Transformers 4.40.2
 - Pytorch 2.1.2+cu118
+- Datasets 2.19.1
+- Tokenizers 0.19.1

all_results.json CHANGED Viewed

@@ -1,8 +1,9 @@
 {
     "epoch": 1.0,
-    "train_loss": 0.44703073270859256,
-    "train_runtime": 13842.8667,
-    "train_samples": 111134,
-    "train_samples_per_second": 8.028,
     "train_steps_per_second": 0.063
 }

 {
     "epoch": 1.0,
+    "total_flos": 0.0,
+    "train_loss": 0.4007269041922391,
+    "train_runtime": 6210.4356,
+    "train_samples": 50000,
+    "train_samples_per_second": 8.051,
     "train_steps_per_second": 0.063
 }

config.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "/mnt/bn/xuruijie-llm/checkpoints/hh-rlhf/sft_0521/checkpoint-5500/",
   "architectures": [
     "MistralForCausalLM"
   ],
@@ -20,7 +20,7 @@
   "sliding_window": 4096,
   "tie_word_embeddings": false,
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.41.1",
   "use_cache": false,
   "vocab_size": 32002
 }

 {
+  "_name_or_path": "/mnt/bn/xuruijie-llm/checkpoints/new_world/v1-ultral",
   "architectures": [
     "MistralForCausalLM"
   ],
   "sliding_window": 4096,
   "tie_word_embeddings": false,
   "torch_dtype": "bfloat16",
+  "transformers_version": "4.40.2",
   "use_cache": false,
   "vocab_size": 32002
 }

generation_config.json CHANGED Viewed

@@ -2,5 +2,5 @@
   "_from_model_config": true,
   "bos_token_id": 1,
   "eos_token_id": 32000,
-  "transformers_version": "4.38.2"
 }

   "_from_model_config": true,
   "bos_token_id": 1,
   "eos_token_id": 32000,
+  "transformers_version": "4.40.2"
 }

model-00001-of-00003.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6be203077af844af5de32b657bbc2702f49a7f3828bee12a42b12c1a74280723
 size 4943178720

 version https://git-lfs.github.com/spec/v1
+oid sha256:a9f251e96294c9492756c37d895155f82d849066c2471fd0aca8729a7aede122
 size 4943178720

model-00002-of-00003.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:40ddbab26b4c540c98068ca370e93ec6d4c67b3e6238f0b4926eb9fc2f1596b5
 size 4999819336

 version https://git-lfs.github.com/spec/v1
+oid sha256:9e9fd416637a101fc45a516e2d108838af2259d6ce2b49a13aba516cf189b7f0
 size 4999819336

model-00003-of-00003.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:76160dd669972969580331cecaeb5b1949bff1c3100a65adafe8e5985f40204c
 size 4540532728

 version https://git-lfs.github.com/spec/v1
+oid sha256:551b3715b02a97fd2af4d39745e82e183fc10809e578801af9fb5fdd39551e6f
 size 4540532728

runs/May28_00-07-25_n136-112-146/events.out.tfevents.1716826553.n136-112-146.284037.0 CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d00d7a6f35edb4776ffcc0381341f6e0a886f5ba1c80665a4f893b2276a86a7c
-size 28347

 version https://git-lfs.github.com/spec/v1
+oid sha256:96385d3b27773ea70ec55132d40a85c9739d029bd06e376948e2a8e29febfd37
+size 34893

train_results.json CHANGED Viewed

@@ -1,8 +1,9 @@
 {
     "epoch": 1.0,
-    "train_loss": 0.44703073270859256,
-    "train_runtime": 13842.8667,
-    "train_samples": 111134,
-    "train_samples_per_second": 8.028,
     "train_steps_per_second": 0.063
 }

 {
     "epoch": 1.0,
+    "total_flos": 0.0,
+    "train_loss": 0.4007269041922391,
+    "train_runtime": 6210.4356,
+    "train_samples": 50000,
+    "train_samples_per_second": 8.051,
     "train_steps_per_second": 0.063
 }

trainer_state.json CHANGED Viewed

@@ -1,21 +1,21 @@
 {
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 0.9994242947610823,
   "eval_steps": 100,
-  "global_step": 868,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
-      "epoch": 0.0,
-      "grad_norm": 35.129907946655834,
-      "learning_rate": 5.747126436781609e-09,
-      "logits/chosen": -1.8631134033203125,
-      "logits/rejected": -1.9713879823684692,
-      "logps/chosen": -395.93560791015625,
-      "logps/rejected": -290.8868408203125,
       "loss": 0.6931,
       "rewards/accuracies": 0.0,
       "rewards/chosen": 0.0,
@@ -24,1435 +24,650 @@
       "step": 1
     },
     {
-      "epoch": 0.01,
-      "grad_norm": 38.80010312606095,
-      "learning_rate": 5.747126436781609e-08,
-      "logits/chosen": -2.041348457336426,
-      "logits/rejected": -1.9895708560943604,
-      "logps/chosen": -276.6951599121094,
-      "logps/rejected": -224.53475952148438,
-      "loss": 0.6932,
-      "rewards/accuracies": 0.5,
-      "rewards/chosen": 0.000690977496560663,
-      "rewards/margins": 0.00028596253832802176,
-      "rewards/rejected": 0.00040501501644030213,
       "step": 10
     },
     {
-      "epoch": 0.02,
-      "grad_norm": 34.46172746061437,
-      "learning_rate": 1.1494252873563217e-07,
-      "logits/chosen": -2.1249823570251465,
-      "logits/rejected": -2.0916128158569336,
-      "logps/chosen": -274.7043151855469,
-      "logps/rejected": -239.5059051513672,
-      "loss": 0.692,
-      "rewards/accuracies": 0.5562499761581421,
-      "rewards/chosen": 0.004372459836304188,
-      "rewards/margins": 0.002485192846506834,
-      "rewards/rejected": 0.0018872671062126756,
       "step": 20
     },
     {
-      "epoch": 0.03,
-      "grad_norm": 33.511897886478806,
-      "learning_rate": 1.7241379310344828e-07,
-      "logits/chosen": -2.1329550743103027,
-      "logits/rejected": -2.1377687454223633,
-      "logps/chosen": -245.65478515625,
-      "logps/rejected": -219.1451873779297,
-      "loss": 0.6874,
-      "rewards/accuracies": 0.612500011920929,
-      "rewards/chosen": 0.025536498054862022,
-      "rewards/margins": 0.010243075899779797,
-      "rewards/rejected": 0.015293421223759651,
       "step": 30
     },
     {
-      "epoch": 0.05,
-      "grad_norm": 30.025842330171532,
-      "learning_rate": 2.2988505747126435e-07,
-      "logits/chosen": -2.1390576362609863,
-      "logits/rejected": -2.130236864089966,
-      "logps/chosen": -259.17510986328125,
-      "logps/rejected": -241.2788848876953,
-      "loss": 0.676,
-      "rewards/accuracies": 0.6937500238418579,
-      "rewards/chosen": 0.08977474272251129,
-      "rewards/margins": 0.036073412746191025,
-      "rewards/rejected": 0.05370132252573967,
       "step": 40
     },
     {
-      "epoch": 0.06,
-      "grad_norm": 26.283273801474603,
-      "learning_rate": 2.873563218390804e-07,
-      "logits/chosen": -2.1333060264587402,
-      "logits/rejected": -2.1513326168060303,
-      "logps/chosen": -234.34951782226562,
-      "logps/rejected": -225.25820922851562,
-      "loss": 0.6593,
-      "rewards/accuracies": 0.71875,
-      "rewards/chosen": 0.14864465594291687,
-      "rewards/margins": 0.07653863728046417,
-      "rewards/rejected": 0.0721060186624527,
       "step": 50
     },
     {
-      "epoch": 0.07,
-      "grad_norm": 25.615307740380704,
-      "learning_rate": 3.4482758620689656e-07,
-      "logits/chosen": -2.084411859512329,
-      "logits/rejected": -2.1329734325408936,
-      "logps/chosen": -269.8699035644531,
-      "logps/rejected": -241.92623901367188,
-      "loss": 0.6364,
-      "rewards/accuracies": 0.762499988079071,
-      "rewards/chosen": 0.24832992255687714,
-      "rewards/margins": 0.15597295761108398,
-      "rewards/rejected": 0.09235697984695435,
       "step": 60
     },
     {
-      "epoch": 0.08,
-      "grad_norm": 24.88575351107564,
-      "learning_rate": 4.0229885057471266e-07,
-      "logits/chosen": -2.2113332748413086,
-      "logits/rejected": -2.2210419178009033,
-      "logps/chosen": -264.09210205078125,
-      "logps/rejected": -247.2978515625,
-      "loss": 0.6202,
-      "rewards/accuracies": 0.7749999761581421,
-      "rewards/chosen": 0.18283841013908386,
-      "rewards/margins": 0.1584111750125885,
-      "rewards/rejected": 0.024427231401205063,
       "step": 70
     },
     {
-      "epoch": 0.09,
-      "grad_norm": 27.297246578083513,
-      "learning_rate": 4.597701149425287e-07,
-      "logits/chosen": -2.150770425796509,
-      "logits/rejected": -2.1732804775238037,
-      "logps/chosen": -274.4136962890625,
-      "logps/rejected": -255.2479248046875,
-      "loss": 0.5908,
-      "rewards/accuracies": 0.768750011920929,
-      "rewards/chosen": 0.08161883056163788,
-      "rewards/margins": 0.29927313327789307,
-      "rewards/rejected": -0.2176542729139328,
       "step": 80
     },
     {
-      "epoch": 0.1,
-      "grad_norm": 28.084151245430267,
-      "learning_rate": 4.999817969178237e-07,
-      "logits/chosen": -2.112828254699707,
-      "logits/rejected": -2.1538023948669434,
-      "logps/chosen": -304.37359619140625,
-      "logps/rejected": -301.63006591796875,
-      "loss": 0.5668,
-      "rewards/accuracies": 0.7749999761581421,
-      "rewards/chosen": -0.11925234645605087,
-      "rewards/margins": 0.42082518339157104,
-      "rewards/rejected": -0.5400775074958801,
       "step": 90
     },
     {
-      "epoch": 0.12,
-      "grad_norm": 31.821457063829286,
-      "learning_rate": 4.996582603056428e-07,
-      "logits/chosen": -2.128788948059082,
-      "logits/rejected": -2.174410581588745,
-      "logps/chosen": -313.7573547363281,
-      "logps/rejected": -319.2353820800781,
-      "loss": 0.5557,
-      "rewards/accuracies": 0.7250000238418579,
-      "rewards/chosen": -0.30633050203323364,
-      "rewards/margins": 0.45107221603393555,
-      "rewards/rejected": -0.7574027180671692,
       "step": 100
     },
     {
-      "epoch": 0.12,
-      "eval_logits/chosen": -2.159374713897705,
-      "eval_logits/rejected": -2.1569156646728516,
-      "eval_logps/chosen": -415.816650390625,
-      "eval_logps/rejected": -445.2613525390625,
-      "eval_loss": 0.6118258833885193,
-      "eval_rewards/accuracies": 0.671875,
-      "eval_rewards/chosen": -0.36439263820648193,
-      "eval_rewards/margins": 0.36210644245147705,
-      "eval_rewards/rejected": -0.7264990210533142,
-      "eval_runtime": 97.4725,
-      "eval_samples_per_second": 20.519,
-      "eval_steps_per_second": 0.328,
       "step": 100
     },
     {
-      "epoch": 0.13,
-      "grad_norm": 36.68008853886969,
-      "learning_rate": 4.989308132738126e-07,
-      "logits/chosen": -2.0658252239227295,
-      "logits/rejected": -2.0949950218200684,
-      "logps/chosen": -320.2489013671875,
-      "logps/rejected": -331.5747375488281,
-      "loss": 0.5429,
-      "rewards/accuracies": 0.75,
-      "rewards/chosen": -0.4602568745613098,
-      "rewards/margins": 0.5536943674087524,
-      "rewards/rejected": -1.013951301574707,
       "step": 110
     },
     {
-      "epoch": 0.14,
-      "grad_norm": 33.72936391730864,
-      "learning_rate": 4.978006327248536e-07,
-      "logits/chosen": -1.987633466720581,
-      "logits/rejected": -1.984405755996704,
-      "logps/chosen": -354.09942626953125,
-      "logps/rejected": -387.8695068359375,
-      "loss": 0.5007,
-      "rewards/accuracies": 0.768750011920929,
-      "rewards/chosen": -0.6619850993156433,
-      "rewards/margins": 0.6314666271209717,
-      "rewards/rejected": -1.2934516668319702,
       "step": 120
     },
     {
-      "epoch": 0.15,
-      "grad_norm": 56.97154479868482,
-      "learning_rate": 4.962695471250032e-07,
-      "logits/chosen": -1.9375616312026978,
-      "logits/rejected": -1.9765644073486328,
-      "logps/chosen": -340.1041564941406,
-      "logps/rejected": -322.09417724609375,
-      "loss": 0.5522,
-      "rewards/accuracies": 0.7562500238418579,
-      "rewards/chosen": -0.7063130140304565,
-      "rewards/margins": 0.4115590453147888,
-      "rewards/rejected": -1.1178721189498901,
       "step": 130
     },
     {
-      "epoch": 0.16,
-      "grad_norm": 37.7377259153511,
-      "learning_rate": 4.94340033546025e-07,
-      "logits/chosen": -2.1993556022644043,
-      "logits/rejected": -2.185791492462158,
-      "logps/chosen": -320.29559326171875,
-      "logps/rejected": -332.7876281738281,
-      "loss": 0.5222,
-      "rewards/accuracies": 0.7749999761581421,
-      "rewards/chosen": -0.33287590742111206,
-      "rewards/margins": 0.6350983381271362,
-      "rewards/rejected": -0.9679743051528931,
       "step": 140
     },
     {
-      "epoch": 0.17,
-      "grad_norm": 35.68760199609404,
-      "learning_rate": 4.920152136576705e-07,
-      "logits/chosen": -2.324455738067627,
-      "logits/rejected": -2.3316075801849365,
-      "logps/chosen": -342.5655517578125,
-      "logps/rejected": -415.856689453125,
-      "loss": 0.4726,
-      "rewards/accuracies": 0.800000011920929,
-      "rewards/chosen": -0.7312036156654358,
-      "rewards/margins": 0.9272117614746094,
-      "rewards/rejected": -1.65841543674469,
       "step": 150
     },
     {
-      "epoch": 0.18,
-      "grad_norm": 37.470403244655515,
-      "learning_rate": 4.892988486772756e-07,
-      "logits/chosen": -2.515658140182495,
-      "logits/rejected": -2.526738405227661,
-      "logps/chosen": -348.3170471191406,
-      "logps/rejected": -359.30633544921875,
-      "loss": 0.5029,
-      "rewards/accuracies": 0.768750011920929,
-      "rewards/chosen": -0.5424179434776306,
-      "rewards/margins": 0.6865107417106628,
-      "rewards/rejected": -1.228928804397583,
       "step": 160
     },
     {
-      "epoch": 0.2,
-      "grad_norm": 45.91976760863689,
-      "learning_rate": 4.861953332846629e-07,
-      "logits/chosen": -2.555476665496826,
-      "logits/rejected": -2.5736944675445557,
-      "logps/chosen": -318.83306884765625,
-      "logps/rejected": -354.2410583496094,
-      "loss": 0.4971,
-      "rewards/accuracies": 0.762499988079071,
-      "rewards/chosen": -0.5986114740371704,
-      "rewards/margins": 0.7181976437568665,
-      "rewards/rejected": -1.3168091773986816,
       "step": 170
     },
     {
-      "epoch": 0.21,
-      "grad_norm": 39.45147614462311,
-      "learning_rate": 4.827096885121953e-07,
-      "logits/chosen": -2.725339412689209,
-      "logits/rejected": -2.790322780609131,
-      "logps/chosen": -317.9709777832031,
-      "logps/rejected": -369.8156433105469,
-      "loss": 0.4663,
-      "rewards/accuracies": 0.8062499761581421,
-      "rewards/chosen": -0.6137017607688904,
-      "rewards/margins": 0.9897038340568542,
-      "rewards/rejected": -1.6034053564071655,
       "step": 180
     },
     {
-      "epoch": 0.22,
-      "grad_norm": 36.629506229246836,
-      "learning_rate": 4.788475536214821e-07,
-      "logits/chosen": -2.7514729499816895,
-      "logits/rejected": -2.8462166786193848,
-      "logps/chosen": -364.15350341796875,
-      "logps/rejected": -426.7947692871094,
-      "loss": 0.4836,
-      "rewards/accuracies": 0.6937500238418579,
-      "rewards/chosen": -1.0376867055892944,
-      "rewards/margins": 0.9606195688247681,
-      "rewards/rejected": -1.9983062744140625,
       "step": 190
     },
     {
-      "epoch": 0.23,
-      "grad_norm": 34.16905422902762,
-      "learning_rate": 4.746151769798818e-07,
-      "logits/chosen": -2.6336653232574463,
-      "logits/rejected": -2.803025722503662,
-      "logps/chosen": -316.927001953125,
-      "logps/rejected": -380.2194519042969,
-      "loss": 0.4689,
-      "rewards/accuracies": 0.831250011920929,
-      "rewards/chosen": -0.6853088140487671,
-      "rewards/margins": 0.9961303472518921,
-      "rewards/rejected": -1.6814391613006592,
       "step": 200
     },
     {
-      "epoch": 0.23,
-      "eval_logits/chosen": -2.857208251953125,
-      "eval_logits/rejected": -2.9278781414031982,
-      "eval_logps/chosen": -428.585205078125,
-      "eval_logps/rejected": -502.7698669433594,
-      "eval_loss": 0.5067635178565979,
-      "eval_rewards/accuracies": 0.75,
-      "eval_rewards/chosen": -0.49207818508148193,
-      "eval_rewards/margins": 0.8095061779022217,
-      "eval_rewards/rejected": -1.3015843629837036,
-      "eval_runtime": 97.5856,
-      "eval_samples_per_second": 20.495,
-      "eval_steps_per_second": 0.328,
       "step": 200
     },
     {
-      "epoch": 0.24,
-      "grad_norm": 37.010315481275924,
-      "learning_rate": 4.7001940595156055e-07,
-      "logits/chosen": -2.86403226852417,
-      "logits/rejected": -2.915546178817749,
-      "logps/chosen": -384.78106689453125,
-      "logps/rejected": -438.4271545410156,
-      "loss": 0.4609,
-      "rewards/accuracies": 0.699999988079071,
-      "rewards/chosen": -0.7915019989013672,
-      "rewards/margins": 0.7616507411003113,
-      "rewards/rejected": -1.5531526803970337,
       "step": 210
     },
     {
-      "epoch": 0.25,
-      "grad_norm": 44.86769250632604,
-      "learning_rate": 4.650676758194623e-07,
-      "logits/chosen": -2.865612268447876,
-      "logits/rejected": -2.9143929481506348,
-      "logps/chosen": -381.2469177246094,
-      "logps/rejected": -441.4414978027344,
-      "loss": 0.4525,
-      "rewards/accuracies": 0.7749999761581421,
-      "rewards/chosen": -0.8893365859985352,
-      "rewards/margins": 1.0212167501449585,
-      "rewards/rejected": -1.910552978515625,
       "step": 220
     },
     {
-      "epoch": 0.26,
-      "grad_norm": 38.712466368217875,
-      "learning_rate": 4.5976799775611215e-07,
-      "logits/chosen": -3.0207266807556152,
-      "logits/rejected": -3.1139583587646484,
-      "logps/chosen": -360.5721130371094,
-      "logps/rejected": -431.66455078125,
-      "loss": 0.4648,
-      "rewards/accuracies": 0.71875,
-      "rewards/chosen": -0.9129144549369812,
-      "rewards/margins": 0.9306351542472839,
-      "rewards/rejected": -1.8435497283935547,
       "step": 230
     },
     {
-      "epoch": 0.28,
-      "grad_norm": 52.80138142568913,
-      "learning_rate": 4.5412894586271543e-07,
-      "logits/chosen": -3.160139322280884,
-      "logits/rejected": -3.2236475944519043,
-      "logps/chosen": -365.31378173828125,
-      "logps/rejected": -439.4544372558594,
-      "loss": 0.4615,
-      "rewards/accuracies": 0.731249988079071,
-      "rewards/chosen": -0.9612631797790527,
-      "rewards/margins": 1.0881351232528687,
-      "rewards/rejected": -2.049398183822632,
       "step": 240
     },
     {
-      "epoch": 0.29,
-      "grad_norm": 32.70960725306299,
-      "learning_rate": 4.481596432975201e-07,
-      "logits/chosen": -3.0666606426239014,
-      "logits/rejected": -3.2191436290740967,
-      "logps/chosen": -358.38006591796875,
-      "logps/rejected": -420.057861328125,
-      "loss": 0.4566,
-      "rewards/accuracies": 0.762499988079071,
-      "rewards/chosen": -0.7017436623573303,
-      "rewards/margins": 0.9394323229789734,
-      "rewards/rejected": -1.6411759853363037,
       "step": 250
     },
     {
-      "epoch": 0.3,
-      "grad_norm": 41.45445866458212,
-      "learning_rate": 4.41869747515886e-07,
-      "logits/chosen": -2.946607828140259,
-      "logits/rejected": -3.0797600746154785,
-      "logps/chosen": -370.9931335449219,
-      "logps/rejected": -431.0896911621094,
-      "loss": 0.4354,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -0.5950635075569153,
-      "rewards/margins": 1.3376449346542358,
-      "rewards/rejected": -1.932708740234375,
       "step": 260
     },
     {
-      "epoch": 0.31,
-      "grad_norm": 88.17109933362657,
-      "learning_rate": 4.352694346459396e-07,
-      "logits/chosen": -2.953822374343872,
-      "logits/rejected": -3.0674452781677246,
-      "logps/chosen": -338.36041259765625,
-      "logps/rejected": -427.46630859375,
-      "loss": 0.4567,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -0.6696688532829285,
-      "rewards/margins": 1.309070348739624,
-      "rewards/rejected": -1.9787391424179077,
       "step": 270
     },
     {
-      "epoch": 0.32,
-      "grad_norm": 34.8827153941344,
-      "learning_rate": 4.2836938302509256e-07,
-      "logits/chosen": -3.1520888805389404,
-      "logits/rejected": -3.3024184703826904,
-      "logps/chosen": -344.01055908203125,
-      "logps/rejected": -420.909423828125,
-      "loss": 0.4396,
-      "rewards/accuracies": 0.8125,
-      "rewards/chosen": -0.8860230445861816,
-      "rewards/margins": 1.1038401126861572,
-      "rewards/rejected": -1.989863395690918,
       "step": 280
     },
     {
-      "epoch": 0.33,
-      "grad_norm": 35.155375356854826,
-      "learning_rate": 4.2118075592405874e-07,
-      "logits/chosen": -3.3648910522460938,
-      "logits/rejected": -3.546940565109253,
-      "logps/chosen": -351.7215881347656,
-      "logps/rejected": -464.04541015625,
-      "loss": 0.4268,
-      "rewards/accuracies": 0.7749999761581421,
-      "rewards/chosen": -1.1209633350372314,
-      "rewards/margins": 1.3907191753387451,
-      "rewards/rejected": -2.5116827487945557,
       "step": 290
     },
     {
-      "epoch": 0.35,
-      "grad_norm": 34.14384163098325,
-      "learning_rate": 4.137151834863213e-07,
-      "logits/chosen": -3.596907377243042,
-      "logits/rejected": -3.7265594005584717,
-      "logps/chosen": -371.98834228515625,
-      "logps/rejected": -488.18975830078125,
-      "loss": 0.4351,
-      "rewards/accuracies": 0.8125,
-      "rewards/chosen": -0.8739571571350098,
-      "rewards/margins": 1.3305376768112183,
-      "rewards/rejected": -2.2044949531555176,
       "step": 300
     },
     {
-      "epoch": 0.35,
-      "eval_logits/chosen": -3.4653797149658203,
-      "eval_logits/rejected": -3.652350425720215,
-      "eval_logps/chosen": -432.01080322265625,
-      "eval_logps/rejected": -524.9551391601562,
-      "eval_loss": 0.4574473202228546,
-      "eval_rewards/accuracies": 0.79296875,
-      "eval_rewards/chosen": -0.5263344645500183,
-      "eval_rewards/margins": 0.9971021413803101,
-      "eval_rewards/rejected": -1.5234365463256836,
-      "eval_runtime": 97.4167,
-      "eval_samples_per_second": 20.53,
-      "eval_steps_per_second": 0.328,
       "step": 300
     },
     {
-      "epoch": 0.36,
-      "grad_norm": 45.59368283329677,
-      "learning_rate": 4.059847439122671e-07,
-      "logits/chosen": -3.523773193359375,
-      "logits/rejected": -3.7083168029785156,
-      "logps/chosen": -374.57525634765625,
-      "logps/rejected": -469.28533935546875,
-      "loss": 0.4288,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -1.021723747253418,
-      "rewards/margins": 1.3690807819366455,
-      "rewards/rejected": -2.3908047676086426,
       "step": 310
     },
     {
-      "epoch": 0.37,
-      "grad_norm": 41.39688664580757,
-      "learning_rate": 3.98001943918432e-07,
-      "logits/chosen": -3.278153896331787,
-      "logits/rejected": -3.5683510303497314,
-      "logps/chosen": -419.21868896484375,
-      "logps/rejected": -439.51837158203125,
-      "loss": 0.4438,
-      "rewards/accuracies": 0.78125,
-      "rewards/chosen": -0.8592584729194641,
-      "rewards/margins": 1.153443694114685,
-      "rewards/rejected": -2.012701988220215,
       "step": 320
     },
     {
-      "epoch": 0.38,
-      "grad_norm": 48.21450790202707,
-      "learning_rate": 3.8977969850346866e-07,
-      "logits/chosen": -3.3799691200256348,
-      "logits/rejected": -3.6289849281311035,
-      "logps/chosen": -297.0557861328125,
-      "logps/rejected": -381.7215270996094,
-      "loss": 0.4422,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -0.5932060480117798,
-      "rewards/margins": 1.224426507949829,
-      "rewards/rejected": -1.8176324367523193,
       "step": 330
     },
     {
-      "epoch": 0.39,
-      "grad_norm": 43.46878894172161,
-      "learning_rate": 3.8133131005357465e-07,
-      "logits/chosen": -3.379418134689331,
-      "logits/rejected": -3.6483395099639893,
-      "logps/chosen": -331.81390380859375,
-      "logps/rejected": -441.87957763671875,
-      "loss": 0.4286,
-      "rewards/accuracies": 0.731249988079071,
-      "rewards/chosen": -0.8891918063163757,
-      "rewards/margins": 1.391618013381958,
-      "rewards/rejected": -2.2808098793029785,
       "step": 340
     },
     {
-      "epoch": 0.4,
-      "grad_norm": 45.9545965651521,
-      "learning_rate": 3.7267044682118435e-07,
-      "logits/chosen": -3.411963939666748,
-      "logits/rejected": -3.587937593460083,
-      "logps/chosen": -346.03521728515625,
-      "logps/rejected": -445.9239807128906,
-      "loss": 0.4628,
-      "rewards/accuracies": 0.768750011920929,
-      "rewards/chosen": -0.8978070020675659,
-      "rewards/margins": 1.216806173324585,
-      "rewards/rejected": -2.1146132946014404,
       "step": 350
     },
     {
-      "epoch": 0.41,
-      "grad_norm": 46.21389082842024,
-      "learning_rate": 3.638111208117425e-07,
-      "logits/chosen": -3.389965772628784,
-      "logits/rejected": -3.6526336669921875,
-      "logps/chosen": -344.18798828125,
-      "logps/rejected": -436.8316955566406,
-      "loss": 0.4428,
-      "rewards/accuracies": 0.78125,
-      "rewards/chosen": -0.7493476867675781,
-      "rewards/margins": 1.355184555053711,
-      "rewards/rejected": -2.10453200340271,
       "step": 360
     },
     {
-      "epoch": 0.43,
-      "grad_norm": 36.698900906534384,
-      "learning_rate": 3.5476766511433605e-07,
-      "logits/chosen": -3.333527088165283,
-      "logits/rejected": -3.72214937210083,
-      "logps/chosen": -369.4108581542969,
-      "logps/rejected": -476.28759765625,
-      "loss": 0.4406,
-      "rewards/accuracies": 0.762499988079071,
-      "rewards/chosen": -0.7729538679122925,
-      "rewards/margins": 1.6176202297210693,
-      "rewards/rejected": -2.3905742168426514,
       "step": 370
     },
     {
-      "epoch": 0.44,
-      "grad_norm": 36.19981517638812,
-      "learning_rate": 3.455547107128602e-07,
-      "logits/chosen": -3.6403069496154785,
-      "logits/rejected": -3.8383822441101074,
-      "logps/chosen": -354.4368591308594,
-      "logps/rejected": -423.34710693359375,
-      "loss": 0.4292,
-      "rewards/accuracies": 0.824999988079071,
-      "rewards/chosen": -0.5367406606674194,
-      "rewards/margins": 1.240304946899414,
-      "rewards/rejected": -1.777045488357544,
       "step": 380
     },
     {
-      "epoch": 0.45,
-      "grad_norm": 46.389830142880655,
-      "learning_rate": 3.361871628152338e-07,
-      "logits/chosen": -3.556725263595581,
-      "logits/rejected": -3.9898459911346436,
-      "logps/chosen": -356.79791259765625,
-      "logps/rejected": -472.49267578125,
-      "loss": 0.4147,
-      "rewards/accuracies": 0.793749988079071,
-      "rewards/chosen": -0.7947701811790466,
-      "rewards/margins": 1.4320166110992432,
-      "rewards/rejected": -2.2267866134643555,
       "step": 390
     },
-    {
-      "epoch": 0.46,
-      "grad_norm": 44.843204651815924,
-      "learning_rate": 3.2668017673896077e-07,
-      "logits/chosen": -3.4979865550994873,
-      "logits/rejected": -3.931288957595825,
-      "logps/chosen": -354.89971923828125,
-      "logps/rejected": -465.4832458496094,
-      "loss": 0.3978,
-      "rewards/accuracies": 0.762499988079071,
-      "rewards/chosen": -0.9876173138618469,
-      "rewards/margins": 1.556265115737915,
-      "rewards/rejected": -2.543882369995117,
-      "step": 400
-    },
-    {
-      "epoch": 0.46,
-      "eval_logits/chosen": -3.5189661979675293,
-      "eval_logits/rejected": -3.8427517414093018,
-      "eval_logps/chosen": -431.57208251953125,
-      "eval_logps/rejected": -545.3043823242188,
-      "eval_loss": 0.4129987061023712,
-      "eval_rewards/accuracies": 0.8359375,
-      "eval_rewards/chosen": -0.5219463109970093,
-      "eval_rewards/margins": 1.204982876777649,
-      "eval_rewards/rejected": -1.7269293069839478,
-      "eval_runtime": 97.4385,
-      "eval_samples_per_second": 20.526,
-      "eval_steps_per_second": 0.328,
-      "step": 400
-    },
-    {
-      "epoch": 0.47,
-      "grad_norm": 47.76955752807069,
-      "learning_rate": 3.1704913339205103e-07,
-      "logits/chosen": -3.461193799972534,
-      "logits/rejected": -3.710247755050659,
-      "logps/chosen": -387.45135498046875,
-      "logps/rejected": -511.35479736328125,
-      "loss": 0.4208,
-      "rewards/accuracies": 0.762499988079071,
-      "rewards/chosen": -0.8673557043075562,
-      "rewards/margins": 1.5246984958648682,
-      "rewards/rejected": -2.392054319381714,
-      "step": 410
-    },
-    {
-      "epoch": 0.48,
-      "grad_norm": 39.876214448639374,
-      "learning_rate": 3.0730961438896885e-07,
-      "logits/chosen": -3.5806972980499268,
-      "logits/rejected": -3.978752613067627,
-      "logps/chosen": -361.4602966308594,
-      "logps/rejected": -514.3541870117188,
-      "loss": 0.4073,
-      "rewards/accuracies": 0.8500000238418579,
-      "rewards/chosen": -0.8498995900154114,
-      "rewards/margins": 1.8191858530044556,
-      "rewards/rejected": -2.6690852642059326,
-      "step": 420
-    },
-    {
-      "epoch": 0.5,
-      "grad_norm": 38.614610181092246,
-      "learning_rate": 2.9747737684186795e-07,
-      "logits/chosen": -3.636990785598755,
-      "logits/rejected": -3.9850502014160156,
-      "logps/chosen": -363.47320556640625,
-      "logps/rejected": -452.01422119140625,
-      "loss": 0.428,
-      "rewards/accuracies": 0.7749999761581421,
-      "rewards/chosen": -0.8123588562011719,
-      "rewards/margins": 1.3265635967254639,
-      "rewards/rejected": -2.138922691345215,
-      "step": 430
-    },
-    {
-      "epoch": 0.51,
-      "grad_norm": 42.82087860142725,
-      "learning_rate": 2.8756832786789663e-07,
-      "logits/chosen": -3.677544116973877,
-      "logits/rejected": -4.333060264587402,
-      "logps/chosen": -382.77850341796875,
-      "logps/rejected": -544.0916748046875,
-      "loss": 0.4003,
-      "rewards/accuracies": 0.84375,
-      "rewards/chosen": -0.8904153108596802,
-      "rewards/margins": 2.0021920204162598,
-      "rewards/rejected": -2.8926072120666504,
-      "step": 440
-    },
-    {
-      "epoch": 0.52,
-      "grad_norm": 38.24345657824867,
-      "learning_rate": 2.7759849885381747e-07,
-      "logits/chosen": -3.5210156440734863,
-      "logits/rejected": -4.05521297454834,
-      "logps/chosen": -353.104736328125,
-      "logps/rejected": -479.227294921875,
-      "loss": 0.4046,
-      "rewards/accuracies": 0.7875000238418579,
-      "rewards/chosen": -0.8441875576972961,
-      "rewards/margins": 1.6644071340560913,
-      "rewards/rejected": -2.508594512939453,
-      "step": 450
-    },
-    {
-      "epoch": 0.53,
-      "grad_norm": 45.50414470118614,
-      "learning_rate": 2.675840195195762e-07,
-      "logits/chosen": -3.977940320968628,
-      "logits/rejected": -4.435048580169678,
-      "logps/chosen": -329.555419921875,
-      "logps/rejected": -462.91546630859375,
-      "loss": 0.4234,
-      "rewards/accuracies": 0.800000011920929,
-      "rewards/chosen": -0.9291930198669434,
-      "rewards/margins": 1.639059066772461,
-      "rewards/rejected": -2.568251609802246,
-      "step": 460
-    },
-    {
-      "epoch": 0.54,
-      "grad_norm": 35.15027402538317,
-      "learning_rate": 2.575410918227829e-07,
-      "logits/chosen": -3.773422956466675,
-      "logits/rejected": -4.276736736297607,
-      "logps/chosen": -353.8682556152344,
-      "logps/rejected": -494.83465576171875,
-      "loss": 0.4037,
-      "rewards/accuracies": 0.8125,
-      "rewards/chosen": -0.9828575253486633,
-      "rewards/margins": 1.6659753322601318,
-      "rewards/rejected": -2.6488327980041504,
-      "step": 470
-    },
-    {
-      "epoch": 0.55,
-      "grad_norm": 39.80925602363646,
-      "learning_rate": 2.474859637463226e-07,
-      "logits/chosen": -3.872162342071533,
-      "logits/rejected": -4.276012897491455,
-      "logps/chosen": -351.5787048339844,
-      "logps/rejected": -477.1171875,
-      "loss": 0.4087,
-      "rewards/accuracies": 0.8500000238418579,
-      "rewards/chosen": -0.8465379476547241,
-      "rewards/margins": 1.8354904651641846,
-      "rewards/rejected": -2.682028293609619,
-      "step": 480
-    },
-    {
-      "epoch": 0.56,
-      "grad_norm": 42.97505867432159,
-      "learning_rate": 2.3743490301150355e-07,
-      "logits/chosen": -3.802455186843872,
-      "logits/rejected": -4.2998151779174805,
-      "logps/chosen": -407.70086669921875,
-      "logps/rejected": -548.3660888671875,
-      "loss": 0.4037,
-      "rewards/accuracies": 0.8062499761581421,
-      "rewards/chosen": -1.167244553565979,
-      "rewards/margins": 1.6480038166046143,
-      "rewards/rejected": -2.8152482509613037,
-      "step": 490
-    },
-    {
-      "epoch": 0.58,
-      "grad_norm": 42.20357617999531,
-      "learning_rate": 2.274041707592724e-07,
-      "logits/chosen": -3.8027241230010986,
-      "logits/rejected": -4.319924831390381,
-      "logps/chosen": -367.37005615234375,
-      "logps/rejected": -488.4521484375,
-      "loss": 0.422,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -1.0884382724761963,
-      "rewards/margins": 1.6187102794647217,
-      "rewards/rejected": -2.707148551940918,
-      "step": 500
-    },
-    {
-      "epoch": 0.58,
-      "eval_logits/chosen": -3.665224552154541,
-      "eval_logits/rejected": -3.9749138355255127,
-      "eval_logps/chosen": -432.22039794921875,
-      "eval_logps/rejected": -549.4501953125,
-      "eval_loss": 0.3803901970386505,
-      "eval_rewards/accuracies": 0.8515625,
-      "eval_rewards/chosen": -0.5284299254417419,
-      "eval_rewards/margins": 1.2399569749832153,
-      "eval_rewards/rejected": -1.7683868408203125,
-      "eval_runtime": 97.3437,
-      "eval_samples_per_second": 20.546,
-      "eval_steps_per_second": 0.329,
-      "step": 500
-    },
-    {
-      "epoch": 0.59,
-      "grad_norm": 33.051939245132566,
-      "learning_rate": 2.17409995242075e-07,
-      "logits/chosen": -3.456348419189453,
-      "logits/rejected": -3.8555312156677246,
-      "logps/chosen": -373.09674072265625,
-      "logps/rejected": -477.450927734375,
-      "loss": 0.4091,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -0.6009220480918884,
-      "rewards/margins": 1.515817403793335,
-      "rewards/rejected": -2.116739511489868,
-      "step": 510
-    },
-    {
-      "epoch": 0.6,
-      "grad_norm": 41.383478497964475,
-      "learning_rate": 2.0746854556892544e-07,
-      "logits/chosen": -3.676243305206299,
-      "logits/rejected": -4.275219917297363,
-      "logps/chosen": -366.2404479980469,
-      "logps/rejected": -504.5061950683594,
-      "loss": 0.4012,
-      "rewards/accuracies": 0.800000011920929,
-      "rewards/chosen": -0.9555729627609253,
-      "rewards/margins": 1.7532062530517578,
-      "rewards/rejected": -2.7087793350219727,
-      "step": 520
-    },
-    {
-      "epoch": 0.61,
-      "grad_norm": 44.14308194161755,
-      "learning_rate": 1.9759590554616173e-07,
-      "logits/chosen": -3.820181369781494,
-      "logits/rejected": -4.501372814178467,
-      "logps/chosen": -359.88836669921875,
-      "logps/rejected": -508.40576171875,
-      "loss": 0.3817,
-      "rewards/accuracies": 0.856249988079071,
-      "rewards/chosen": -1.0667827129364014,
-      "rewards/margins": 1.8904939889907837,
-      "rewards/rejected": -2.9572768211364746,
-      "step": 530
-    },
-    {
-      "epoch": 0.62,
-      "grad_norm": 39.56708257885465,
-      "learning_rate": 1.8780804765620746e-07,
-      "logits/chosen": -3.6856751441955566,
-      "logits/rejected": -4.133795738220215,
-      "logps/chosen": -404.65997314453125,
-      "logps/rejected": -513.8917236328125,
-      "loss": 0.3999,
-      "rewards/accuracies": 0.78125,
-      "rewards/chosen": -1.0721806287765503,
-      "rewards/margins": 1.6528995037078857,
-      "rewards/rejected": -2.7250800132751465,
-      "step": 540
-    },
-    {
-      "epoch": 0.63,
-      "grad_norm": 44.047245390890936,
-      "learning_rate": 1.7812080721643973e-07,
-      "logits/chosen": -3.6747565269470215,
-      "logits/rejected": -4.137899398803711,
-      "logps/chosen": -376.62615966796875,
-      "logps/rejected": -503.29266357421875,
-      "loss": 0.4151,
-      "rewards/accuracies": 0.800000011920929,
-      "rewards/chosen": -0.8548510670661926,
-      "rewards/margins": 1.795414686203003,
-      "rewards/rejected": -2.650266170501709,
-      "step": 550
-    },
-    {
-      "epoch": 0.64,
-      "grad_norm": 44.733238192011385,
-      "learning_rate": 1.6854985675997063e-07,
-      "logits/chosen": -3.412196397781372,
-      "logits/rejected": -3.9336624145507812,
-      "logps/chosen": -365.46795654296875,
-      "logps/rejected": -478.50177001953125,
-      "loss": 0.415,
-      "rewards/accuracies": 0.793749988079071,
-      "rewards/chosen": -0.848841667175293,
-      "rewards/margins": 1.497734785079956,
-      "rewards/rejected": -2.346576690673828,
-      "step": 560
-    },
-    {
-      "epoch": 0.66,
-      "grad_norm": 50.78656478745754,
-      "learning_rate": 1.5911068067978818e-07,
-      "logits/chosen": -3.570014476776123,
-      "logits/rejected": -3.9861984252929688,
-      "logps/chosen": -386.4393005371094,
-      "logps/rejected": -537.798095703125,
-      "loss": 0.3867,
-      "rewards/accuracies": 0.831250011920929,
-      "rewards/chosen": -0.9028172492980957,
-      "rewards/margins": 1.7015517950057983,
-      "rewards/rejected": -2.6043689250946045,
-      "step": 570
-    },
-    {
-      "epoch": 0.67,
-      "grad_norm": 37.91828473174147,
-      "learning_rate": 1.4981855017728197e-07,
-      "logits/chosen": -3.787682294845581,
-      "logits/rejected": -4.240976333618164,
-      "logps/chosen": -353.59393310546875,
-      "logps/rejected": -502.26544189453125,
-      "loss": 0.3785,
-      "rewards/accuracies": 0.831250011920929,
-      "rewards/chosen": -1.1089229583740234,
-      "rewards/margins": 1.7554630041122437,
-      "rewards/rejected": -2.8643860816955566,
-      "step": 580
-    },
-    {
-      "epoch": 0.68,
-      "grad_norm": 41.30885920488077,
-      "learning_rate": 1.406884985556804e-07,
-      "logits/chosen": -3.8037681579589844,
-      "logits/rejected": -4.2103681564331055,
-      "logps/chosen": -368.0989685058594,
-      "logps/rejected": -479.09893798828125,
-      "loss": 0.4082,
-      "rewards/accuracies": 0.800000011920929,
-      "rewards/chosen": -0.9101566076278687,
-      "rewards/margins": 1.4272234439849854,
-      "rewards/rejected": -2.3373799324035645,
-      "step": 590
-    },
-    {
-      "epoch": 0.69,
-      "grad_norm": 33.201388393145585,
-      "learning_rate": 1.3173529689837354e-07,
-      "logits/chosen": -3.958705186843872,
-      "logits/rejected": -4.450201988220215,
-      "logps/chosen": -338.07318115234375,
-      "logps/rejected": -459.9598083496094,
-      "loss": 0.3728,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -0.8850952386856079,
-      "rewards/margins": 1.6357240676879883,
-      "rewards/rejected": -2.5208191871643066,
-      "step": 600
-    },
-    {
-      "epoch": 0.69,
-      "eval_logits/chosen": -3.9204437732696533,
-      "eval_logits/rejected": -4.349194526672363,
-      "eval_logps/chosen": -447.3841857910156,
-      "eval_logps/rejected": -581.492919921875,
-      "eval_loss": 0.3498460352420807,
-      "eval_rewards/accuracies": 0.85546875,
-      "eval_rewards/chosen": -0.6800678968429565,
-      "eval_rewards/margins": 1.408746600151062,
-      "eval_rewards/rejected": -2.0888142585754395,
-      "eval_runtime": 97.3846,
-      "eval_samples_per_second": 20.537,
-      "eval_steps_per_second": 0.329,
-      "step": 600
-    },
-    {
-      "epoch": 0.7,
-      "grad_norm": 38.28289884444149,
-      "learning_rate": 1.2297343017146726e-07,
-      "logits/chosen": -3.8682377338409424,
-      "logits/rejected": -4.303684234619141,
-      "logps/chosen": -397.2498474121094,
-      "logps/rejected": -542.2492065429688,
-      "loss": 0.3926,
-      "rewards/accuracies": 0.824999988079071,
-      "rewards/chosen": -1.052696704864502,
-      "rewards/margins": 1.9617723226547241,
-      "rewards/rejected": -3.0144691467285156,
-      "step": 610
-    },
-    {
-      "epoch": 0.71,
-      "grad_norm": 44.09127513083321,
-      "learning_rate": 1.1441707378923474e-07,
-      "logits/chosen": -3.8735458850860596,
-      "logits/rejected": -4.310673713684082,
-      "logps/chosen": -406.3232727050781,
-      "logps/rejected": -514.9022216796875,
-      "loss": 0.3998,
-      "rewards/accuracies": 0.7875000238418579,
-      "rewards/chosen": -1.1163777112960815,
-      "rewards/margins": 1.5480899810791016,
-      "rewards/rejected": -2.664468288421631,
-      "step": 620
-    },
-    {
-      "epoch": 0.73,
-      "grad_norm": 55.66131693545821,
-      "learning_rate": 1.06080070680377e-07,
-      "logits/chosen": -3.904125690460205,
-      "logits/rejected": -4.380388259887695,
-      "logps/chosen": -392.47015380859375,
-      "logps/rejected": -493.78594970703125,
-      "loss": 0.413,
-      "rewards/accuracies": 0.75,
-      "rewards/chosen": -1.1083078384399414,
-      "rewards/margins": 1.3549706935882568,
-      "rewards/rejected": -2.4632785320281982,
-      "step": 630
-    },
-    {
-      "epoch": 0.74,
-      "grad_norm": 47.57857500564574,
-      "learning_rate": 9.797590889219587e-08,
-      "logits/chosen": -3.7111058235168457,
-      "logits/rejected": -4.241570472717285,
-      "logps/chosen": -390.62103271484375,
-      "logps/rejected": -555.2783813476562,
-      "loss": 0.3857,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -0.9739618301391602,
-      "rewards/margins": 1.9729865789413452,
-      "rewards/rejected": -2.946948289871216,
-      "step": 640
-    },
-    {
-      "epoch": 0.75,
-      "grad_norm": 54.021801628642386,
-      "learning_rate": 9.011769976891367e-08,
-      "logits/chosen": -3.948495388031006,
-      "logits/rejected": -4.6028947830200195,
-      "logps/chosen": -348.66473388671875,
-      "logps/rejected": -499.11309814453125,
-      "loss": 0.3966,
-      "rewards/accuracies": 0.862500011920929,
-      "rewards/chosen": -0.8676958084106445,
-      "rewards/margins": 1.9891424179077148,
-      "rewards/rejected": -2.8568382263183594,
-      "step": 650
-    },
-    {
-      "epoch": 0.76,
-      "grad_norm": 42.830226175531045,
-      "learning_rate": 8.251815673944218e-08,
-      "logits/chosen": -3.9013257026672363,
-      "logits/rejected": -4.469305992126465,
-      "logps/chosen": -356.7492370605469,
-      "logps/rejected": -473.14599609375,
-      "loss": 0.3908,
-      "rewards/accuracies": 0.7875000238418579,
-      "rewards/chosen": -1.1369212865829468,
-      "rewards/margins": 1.5198627710342407,
-      "rewards/rejected": -2.6567840576171875,
-      "step": 660
-    },
-    {
-      "epoch": 0.77,
-      "grad_norm": 35.97105604628807,
-      "learning_rate": 7.518957474892148e-08,
-      "logits/chosen": -3.7510364055633545,
-      "logits/rejected": -4.5982208251953125,
-      "logps/chosen": -359.37835693359375,
-      "logps/rejected": -531.6082153320312,
-      "loss": 0.3667,
-      "rewards/accuracies": 0.8374999761581421,
-      "rewards/chosen": -0.8581075668334961,
-      "rewards/margins": 2.218301773071289,
-      "rewards/rejected": -3.076409101486206,
-      "step": 670
-    },
-    {
-      "epoch": 0.78,
-      "grad_norm": 48.90631940415997,
-      "learning_rate": 6.814381036730274e-08,
-      "logits/chosen": -3.967960834503174,
-      "logits/rejected": -4.668499946594238,
-      "logps/chosen": -347.4683837890625,
-      "logps/rejected": -495.95947265625,
-      "loss": 0.4186,
-      "rewards/accuracies": 0.793749988079071,
-      "rewards/chosen": -1.1356189250946045,
-      "rewards/margins": 1.912573218345642,
-      "rewards/rejected": -3.048192262649536,
-      "step": 680
-    },
-    {
-      "epoch": 0.79,
-      "grad_norm": 44.70641719300435,
-      "learning_rate": 6.139226260715872e-08,
-      "logits/chosen": -3.9364871978759766,
-      "logits/rejected": -4.475349426269531,
-      "logps/chosen": -397.2518310546875,
-      "logps/rejected": -526.1500244140625,
-      "loss": 0.3962,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -1.067858099937439,
-      "rewards/margins": 1.8494539260864258,
-      "rewards/rejected": -2.9173121452331543,
-      "step": 690
-    },
-    {
-      "epoch": 0.81,
-      "grad_norm": 38.79291863118847,
-      "learning_rate": 5.4945854481754734e-08,
-      "logits/chosen": -3.8525912761688232,
-      "logits/rejected": -4.3074188232421875,
-      "logps/chosen": -403.70281982421875,
-      "logps/rejected": -516.3580322265625,
-      "loss": 0.4072,
-      "rewards/accuracies": 0.8062499761581421,
-      "rewards/chosen": -0.8484451174736023,
-      "rewards/margins": 1.8069216012954712,
-      "rewards/rejected": -2.6553666591644287,
-      "step": 700
-    },
-    {
-      "epoch": 0.81,
-      "eval_logits/chosen": -3.8217344284057617,
-      "eval_logits/rejected": -4.235738754272461,
-      "eval_logps/chosen": -438.1347961425781,
-      "eval_logps/rejected": -568.8267211914062,
-      "eval_loss": 0.34126752614974976,
-      "eval_rewards/accuracies": 0.87109375,
-      "eval_rewards/chosen": -0.587573766708374,
-      "eval_rewards/margins": 1.3745783567428589,
-      "eval_rewards/rejected": -1.9621522426605225,
-      "eval_runtime": 97.3651,
-      "eval_samples_per_second": 20.541,
-      "eval_steps_per_second": 0.329,
-      "step": 700
-    },
-    {
-      "epoch": 0.82,
-      "grad_norm": 42.4289491699843,
-      "learning_rate": 4.881501533321605e-08,
-      "logits/chosen": -3.841318130493164,
-      "logits/rejected": -4.349331378936768,
-      "logps/chosen": -369.50439453125,
-      "logps/rejected": -468.9639587402344,
-      "loss": 0.4047,
-      "rewards/accuracies": 0.7875000238418579,
-      "rewards/chosen": -0.9862189292907715,
-      "rewards/margins": 1.4564803838729858,
-      "rewards/rejected": -2.4426989555358887,
-      "step": 710
-    },
-    {
-      "epoch": 0.83,
-      "grad_norm": 34.598541000047966,
-      "learning_rate": 4.300966395938377e-08,
-      "logits/chosen": -3.7932441234588623,
-      "logits/rejected": -4.211984634399414,
-      "logps/chosen": -371.55584716796875,
-      "logps/rejected": -509.62091064453125,
-      "loss": 0.3922,
-      "rewards/accuracies": 0.8500000238418579,
-      "rewards/chosen": -1.0055599212646484,
-      "rewards/margins": 1.8010333776474,
-      "rewards/rejected": -2.806593179702759,
-      "step": 720
-    },
-    {
-      "epoch": 0.84,
-      "grad_norm": 43.981942186199504,
-      "learning_rate": 3.7539192566655246e-08,
-      "logits/chosen": -3.78422212600708,
-      "logits/rejected": -4.4000091552734375,
-      "logps/chosen": -382.2842102050781,
-      "logps/rejected": -524.8851928710938,
-      "loss": 0.3892,
-      "rewards/accuracies": 0.793749988079071,
-      "rewards/chosen": -0.9714164733886719,
-      "rewards/margins": 2.023409128189087,
-      "rewards/rejected": -2.9948253631591797,
-      "step": 730
-    },
-    {
-      "epoch": 0.85,
-      "grad_norm": 47.73160653198726,
-      "learning_rate": 3.24124515747731e-08,
-      "logits/chosen": -3.7954139709472656,
-      "logits/rejected": -4.441324234008789,
-      "logps/chosen": -391.67987060546875,
-      "logps/rejected": -511.4283752441406,
-      "loss": 0.4052,
-      "rewards/accuracies": 0.78125,
-      "rewards/chosen": -1.0643631219863892,
-      "rewards/margins": 1.8877410888671875,
-      "rewards/rejected": -2.952104330062866,
-      "step": 740
-    },
-    {
-      "epoch": 0.86,
-      "grad_norm": 32.946663985678285,
-      "learning_rate": 2.763773529814506e-08,
-      "logits/chosen": -3.753929853439331,
-      "logits/rejected": -4.232656478881836,
-      "logps/chosen": -393.05145263671875,
-      "logps/rejected": -542.4010009765625,
-      "loss": 0.3847,
-      "rewards/accuracies": 0.824999988079071,
-      "rewards/chosen": -1.14042067527771,
-      "rewards/margins": 1.8287441730499268,
-      "rewards/rejected": -2.9691648483276367,
-      "step": 750
-    },
-    {
-      "epoch": 0.88,
-      "grad_norm": 41.44563269413962,
-      "learning_rate": 2.3222768526860698e-08,
-      "logits/chosen": -3.9827494621276855,
-      "logits/rejected": -4.4769439697265625,
-      "logps/chosen": -360.85772705078125,
-      "logps/rejected": -520.6256103515625,
-      "loss": 0.4024,
-      "rewards/accuracies": 0.8374999761581421,
-      "rewards/chosen": -0.9786790013313293,
-      "rewards/margins": 1.8364450931549072,
-      "rewards/rejected": -2.815124034881592,
-      "step": 760
-    },
-    {
-      "epoch": 0.89,
-      "grad_norm": 41.56548534990344,
-      "learning_rate": 1.9174694029115146e-08,
-      "logits/chosen": -3.9031333923339844,
-      "logits/rejected": -4.248320579528809,
-      "logps/chosen": -380.02532958984375,
-      "logps/rejected": -533.628662109375,
-      "loss": 0.3734,
-      "rewards/accuracies": 0.7562500238418579,
-      "rewards/chosen": -1.0660566091537476,
-      "rewards/margins": 1.5903499126434326,
-      "rewards/rejected": -2.6564066410064697,
-      "step": 770
-    },
-    {
-      "epoch": 0.9,
-      "grad_norm": 44.05878651405709,
-      "learning_rate": 1.5500060995258134e-08,
-      "logits/chosen": -3.970170259475708,
-      "logits/rejected": -4.581896781921387,
-      "logps/chosen": -381.94769287109375,
-      "logps/rejected": -505.8392639160156,
-      "loss": 0.3764,
-      "rewards/accuracies": 0.8374999761581421,
-      "rewards/chosen": -1.0537292957305908,
-      "rewards/margins": 1.9894670248031616,
-      "rewards/rejected": -3.043196201324463,
-      "step": 780
-    },
-    {
-      "epoch": 0.91,
-      "grad_norm": 36.69938487477932,
-      "learning_rate": 1.2204814442165812e-08,
-      "logits/chosen": -3.7230095863342285,
-      "logits/rejected": -4.348383903503418,
-      "logps/chosen": -402.3287658691406,
-      "logps/rejected": -528.0650634765625,
-      "loss": 0.3848,
-      "rewards/accuracies": 0.8374999761581421,
-      "rewards/chosen": -0.8664236068725586,
-      "rewards/margins": 1.9163984060287476,
-      "rewards/rejected": -2.7828221321105957,
-      "step": 790
-    },
-    {
-      "epoch": 0.92,
-      "grad_norm": 77.77225540035298,
-      "learning_rate": 9.294285595075669e-09,
-      "logits/chosen": -3.828099012374878,
-      "logits/rejected": -4.454224586486816,
-      "logps/chosen": -401.0879211425781,
-      "logps/rejected": -550.5147705078125,
-      "loss": 0.388,
-      "rewards/accuracies": 0.862500011920929,
-      "rewards/chosen": -1.066781759262085,
-      "rewards/margins": 1.9501686096191406,
-      "rewards/rejected": -3.0169506072998047,
-      "step": 800
-    },
-    {
-      "epoch": 0.92,
-      "eval_logits/chosen": -3.932162046432495,
-      "eval_logits/rejected": -4.366703510284424,
-      "eval_logps/chosen": -442.8078918457031,
-      "eval_logps/rejected": -579.6317138671875,
-      "eval_loss": 0.3309651017189026,
-      "eval_rewards/accuracies": 0.87109375,
-      "eval_rewards/chosen": -0.6343047022819519,
-      "eval_rewards/margins": 1.435897707939148,
-      "eval_rewards/rejected": -2.070202350616455,
-      "eval_runtime": 97.3029,
-      "eval_samples_per_second": 20.554,
-      "eval_steps_per_second": 0.329,
-      "step": 800
-    },
-    {
-      "epoch": 0.93,
-      "grad_norm": 41.84703205105403,
-      "learning_rate": 6.773183262446914e-09,
-      "logits/chosen": -4.019244194030762,
-      "logits/rejected": -4.569401741027832,
-      "logps/chosen": -403.5866394042969,
-      "logps/rejected": -542.3660888671875,
-      "loss": 0.4038,
-      "rewards/accuracies": 0.831250011920929,
-      "rewards/chosen": -1.0097706317901611,
-      "rewards/margins": 1.7525224685668945,
-      "rewards/rejected": -2.7622933387756348,
-      "step": 810
-    },
-    {
-      "epoch": 0.94,
-      "grad_norm": 43.70341312986973,
-      "learning_rate": 4.645586217799452e-09,
-      "logits/chosen": -4.0331034660339355,
-      "logits/rejected": -4.535307884216309,
-      "logps/chosen": -368.22479248046875,
-      "logps/rejected": -495.12615966796875,
-      "loss": 0.4055,
-      "rewards/accuracies": 0.793749988079071,
-      "rewards/chosen": -0.9445031881332397,
-      "rewards/margins": 1.8414275646209717,
-      "rewards/rejected": -2.785930871963501,
-      "step": 820
-    },
-    {
-      "epoch": 0.96,
-      "grad_norm": 39.756368871146144,
-      "learning_rate": 2.9149366008568987e-09,
-      "logits/chosen": -3.8695130348205566,
-      "logits/rejected": -4.424549102783203,
-      "logps/chosen": -382.7538757324219,
-      "logps/rejected": -543.4766845703125,
-      "loss": 0.3869,
-      "rewards/accuracies": 0.831250011920929,
-      "rewards/chosen": -0.9384629130363464,
-      "rewards/margins": 2.0100338459014893,
-      "rewards/rejected": -2.9484963417053223,
-      "step": 830
-    },
-    {
-      "epoch": 0.97,
-      "grad_norm": 41.81245135848943,
-      "learning_rate": 1.5840343486700215e-09,
-      "logits/chosen": -3.9648585319519043,
-      "logits/rejected": -4.646535396575928,
-      "logps/chosen": -393.53857421875,
-      "logps/rejected": -504.774658203125,
-      "loss": 0.385,
-      "rewards/accuracies": 0.8125,
-      "rewards/chosen": -0.9394033551216125,
-      "rewards/margins": 1.7747853994369507,
-      "rewards/rejected": -2.714188814163208,
-      "step": 840
-    },
-    {
-      "epoch": 0.98,
-      "grad_norm": 43.5733624216965,
-      "learning_rate": 6.550326657293881e-10,
-      "logits/chosen": -4.054198265075684,
-      "logits/rejected": -4.520740509033203,
-      "logps/chosen": -346.19580078125,
-      "logps/rejected": -487.6908264160156,
-      "loss": 0.3841,
-      "rewards/accuracies": 0.7749999761581421,
-      "rewards/chosen": -1.0814578533172607,
-      "rewards/margins": 1.7187023162841797,
-      "rewards/rejected": -2.8001601696014404,
-      "step": 850
-    },
-    {
-      "epoch": 0.99,
-      "grad_norm": 41.788009788359076,
-      "learning_rate": 1.2943454039654467e-10,
-      "logits/chosen": -4.023499488830566,
-      "logits/rejected": -4.535801887512207,
-      "logps/chosen": -351.8265686035156,
-      "logps/rejected": -521.2282104492188,
-      "loss": 0.385,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -1.1059845685958862,
-      "rewards/margins": 1.947348952293396,
-      "rewards/rejected": -3.0533337593078613,
-      "step": 860
-    },
     {
       "epoch": 1.0,
-      "step": 868,
       "total_flos": 0.0,
-      "train_loss": 0.44703073270859256,
-      "train_runtime": 13842.8667,
-      "train_samples_per_second": 8.028,
       "train_steps_per_second": 0.063
     }
   ],
   "logging_steps": 10,
-  "max_steps": 868,
   "num_input_tokens_seen": 0,
   "num_train_epochs": 1,
   "save_steps": 100,

 {
   "best_metric": null,
   "best_model_checkpoint": null,
+  "epoch": 1.0,
   "eval_steps": 100,
+  "global_step": 391,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
+      "epoch": 0.0025575447570332483,
+      "grad_norm": 42.05566856949971,
+      "learning_rate": 1.25e-09,
+      "logits/chosen": -4.623842239379883,
+      "logits/rejected": -4.85917854309082,
+      "logps/chosen": -239.31422424316406,
+      "logps/rejected": -207.56365966796875,
       "loss": 0.6931,
       "rewards/accuracies": 0.0,
       "rewards/chosen": 0.0,
       "step": 1
     },
     {
+      "epoch": 0.02557544757033248,
+      "grad_norm": 39.368501883614215,
+      "learning_rate": 1.25e-08,
+      "logits/chosen": -4.334437370300293,
+      "logits/rejected": -4.64446496963501,
+      "logps/chosen": -265.1294250488281,
+      "logps/rejected": -215.75079345703125,
+      "loss": 0.6929,
+      "rewards/accuracies": 0.4513888955116272,
+      "rewards/chosen": -0.00022488113609142601,
+      "rewards/margins": 0.0002015345817198977,
+      "rewards/rejected": -0.0004264157032594085,
       "step": 10
     },
     {
+      "epoch": 0.05115089514066496,
+      "grad_norm": 41.32160817525484,
+      "learning_rate": 2.5e-08,
+      "logits/chosen": -4.507131576538086,
+      "logits/rejected": -4.741620063781738,
+      "logps/chosen": -267.7641906738281,
+      "logps/rejected": -216.6431427001953,
+      "loss": 0.6928,
+      "rewards/accuracies": 0.574999988079071,
+      "rewards/chosen": 0.001757637015543878,
+      "rewards/margins": 0.0020956869702786207,
+      "rewards/rejected": -0.000338050042046234,
       "step": 20
     },
     {
+      "epoch": 0.07672634271099744,
+      "grad_norm": 43.9248767727728,
+      "learning_rate": 3.75e-08,
+      "logits/chosen": -4.58504581451416,
+      "logits/rejected": -4.763764381408691,
+      "logps/chosen": -258.29669189453125,
+      "logps/rejected": -214.74630737304688,
+      "loss": 0.6912,
+      "rewards/accuracies": 0.6312500238418579,
+      "rewards/chosen": 0.00236110738478601,
+      "rewards/margins": 0.0044965180568397045,
+      "rewards/rejected": -0.002135410439223051,
       "step": 30
     },
     {
+      "epoch": 0.10230179028132992,
+      "grad_norm": 43.260460439031476,
+      "learning_rate": 5e-08,
+      "logits/chosen": -4.622879981994629,
+      "logits/rejected": -4.708461284637451,
+      "logps/chosen": -252.55868530273438,
+      "logps/rejected": -220.5706329345703,
+      "loss": 0.6843,
+      "rewards/accuracies": 0.793749988079071,
+      "rewards/chosen": 0.010404979810118675,
+      "rewards/margins": 0.019400382414460182,
+      "rewards/rejected": -0.008995403535664082,
       "step": 40
     },
     {
+      "epoch": 0.1278772378516624,
+      "grad_norm": 42.12222343545266,
+      "learning_rate": 4.989992961303737e-08,
+      "logits/chosen": -4.523015022277832,
+      "logits/rejected": -4.722769737243652,
+      "logps/chosen": -269.7854919433594,
+      "logps/rejected": -228.0283660888672,
+      "loss": 0.6709,
+      "rewards/accuracies": 0.84375,
+      "rewards/chosen": 0.021838409826159477,
+      "rewards/margins": 0.04340698570013046,
+      "rewards/rejected": -0.021568577736616135,
       "step": 50
     },
     {
+      "epoch": 0.1534526854219949,
+      "grad_norm": 41.2743182340779,
+      "learning_rate": 4.960051957873725e-08,
+      "logits/chosen": -4.632551670074463,
+      "logits/rejected": -4.7638654708862305,
+      "logps/chosen": -237.92984008789062,
+      "logps/rejected": -220.9913787841797,
+      "loss": 0.647,
+      "rewards/accuracies": 0.862500011920929,
+      "rewards/chosen": 0.0319262370467186,
+      "rewards/margins": 0.09162561595439911,
+      "rewards/rejected": -0.05969937518239021,
       "step": 60
     },
     {
+      "epoch": 0.17902813299232737,
+      "grad_norm": 39.1856071564541,
+      "learning_rate": 4.910416686333906e-08,
+      "logits/chosen": -4.551320552825928,
+      "logits/rejected": -4.791552543640137,
+      "logps/chosen": -249.8735809326172,
+      "logps/rejected": -230.2266082763672,
+      "loss": 0.6219,
+      "rewards/accuracies": 0.862500011920929,
+      "rewards/chosen": 0.032683782279491425,
+      "rewards/margins": 0.1573253571987152,
+      "rewards/rejected": -0.12464158236980438,
       "step": 70
     },
     {
+      "epoch": 0.20460358056265984,
+      "grad_norm": 40.933579332132446,
+      "learning_rate": 4.841484508350678e-08,
+      "logits/chosen": -4.6027655601501465,
+      "logits/rejected": -4.830323696136475,
+      "logps/chosen": -258.7153625488281,
+      "logps/rejected": -251.62368774414062,
+      "loss": 0.5731,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": 0.025046274065971375,
+      "rewards/margins": 0.2380959540605545,
+      "rewards/rejected": -0.21304969489574432,
       "step": 80
     },
     {
+      "epoch": 0.23017902813299232,
+      "grad_norm": 40.245773651041624,
+      "learning_rate": 4.7538072695020406e-08,
+      "logits/chosen": -4.726979732513428,
+      "logits/rejected": -4.926435470581055,
+      "logps/chosen": -248.79373168945312,
+      "logps/rejected": -241.45919799804688,
+      "loss": 0.5258,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": 0.024190178140997887,
+      "rewards/margins": 0.41048234701156616,
+      "rewards/rejected": -0.38629215955734253,
       "step": 90
     },
     {
+      "epoch": 0.2557544757033248,
+      "grad_norm": 36.66878437086382,
+      "learning_rate": 4.6480868814055416e-08,
+      "logits/chosen": -4.612161636352539,
+      "logits/rejected": -4.8301801681518555,
+      "logps/chosen": -252.9623260498047,
+      "logps/rejected": -275.35980224609375,
+      "loss": 0.5044,
+      "rewards/accuracies": 0.84375,
+      "rewards/chosen": 0.03755917400121689,
+      "rewards/margins": 0.5200667977333069,
+      "rewards/rejected": -0.4825075566768646,
       "step": 100
     },
     {
+      "epoch": 0.2557544757033248,
+      "eval_logits/chosen": -4.655425071716309,
+      "eval_logits/rejected": -4.8306565284729,
+      "eval_logps/chosen": -401.76690673828125,
+      "eval_logps/rejected": -527.1570434570312,
+      "eval_loss": 0.7104954123497009,
+      "eval_rewards/accuracies": 0.47265625,
+      "eval_rewards/chosen": -0.11292455345392227,
+      "eval_rewards/margins": -0.02523117884993553,
+      "eval_rewards/rejected": -0.08769337832927704,
+      "eval_runtime": 98.6531,
+      "eval_samples_per_second": 20.273,
+      "eval_steps_per_second": 0.324,
       "step": 100
     },
     {
+      "epoch": 0.2813299232736573,
+      "grad_norm": 33.07145934836895,
+      "learning_rate": 4.525169702472916e-08,
+      "logits/chosen": -4.64766788482666,
+      "logits/rejected": -4.833477020263672,
+      "logps/chosen": -244.29296875,
+      "logps/rejected": -270.46356201171875,
+      "loss": 0.4684,
+      "rewards/accuracies": 0.856249988079071,
+      "rewards/chosen": 0.04892749339342117,
+      "rewards/margins": 0.586704432964325,
+      "rewards/rejected": -0.5377769470214844,
       "step": 110
     },
     {
+      "epoch": 0.3069053708439898,
+      "grad_norm": 33.5848438881061,
+      "learning_rate": 4.386039762276975e-08,
+      "logits/chosen": -4.567343235015869,
+      "logits/rejected": -4.763147830963135,
+      "logps/chosen": -260.3200988769531,
+      "logps/rejected": -289.777587890625,
+      "loss": 0.4413,
+      "rewards/accuracies": 0.887499988079071,
+      "rewards/chosen": 0.15669476985931396,
+      "rewards/margins": 0.7683843374252319,
+      "rewards/rejected": -0.611689567565918,
       "step": 120
     },
     {
+      "epoch": 0.33248081841432225,
+      "grad_norm": 33.705176699923506,
+      "learning_rate": 4.231810883773999e-08,
+      "logits/chosen": -4.627255916595459,
+      "logits/rejected": -4.849207878112793,
+      "logps/chosen": -243.1725616455078,
+      "logps/rejected": -288.8663635253906,
+      "loss": 0.4074,
+      "rewards/accuracies": 0.893750011920929,
+      "rewards/chosen": 0.15271326899528503,
+      "rewards/margins": 0.8880899548530579,
+      "rewards/rejected": -0.7353767156600952,
       "step": 130
     },
     {
+      "epoch": 0.35805626598465473,
+      "grad_norm": 28.886999424269227,
+      "learning_rate": 4.063717766448194e-08,
+      "logits/chosen": -4.684683322906494,
+      "logits/rejected": -4.887022972106934,
+      "logps/chosen": -271.5019226074219,
+      "logps/rejected": -315.28717041015625,
+      "loss": 0.3858,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": 0.1269315481185913,
+      "rewards/margins": 0.9580682516098022,
+      "rewards/rejected": -0.8311365842819214,
       "step": 140
     },
     {
+      "epoch": 0.3836317135549872,
+      "grad_norm": 34.24616985646881,
+      "learning_rate": 3.8831061017632845e-08,
+      "logits/chosen": -4.733763694763184,
+      "logits/rejected": -4.927274703979492,
+      "logps/chosen": -237.88955688476562,
+      "logps/rejected": -312.8567199707031,
+      "loss": 0.3767,
+      "rewards/accuracies": 0.893750011920929,
+      "rewards/chosen": 0.14528730511665344,
+      "rewards/margins": 1.0826618671417236,
+      "rewards/rejected": -0.937374472618103,
       "step": 150
     },
     {
+      "epoch": 0.4092071611253197,
+      "grad_norm": 32.75808894341811,
+      "learning_rate": 3.691421800053269e-08,
+      "logits/chosen": -4.803037166595459,
+      "logits/rejected": -4.954171657562256,
+      "logps/chosen": -236.3014373779297,
+      "logps/rejected": -314.69329833984375,
+      "loss": 0.3428,
+      "rewards/accuracies": 0.8812500238418579,
+      "rewards/chosen": 0.1515505015850067,
+      "rewards/margins": 1.1423927545547485,
+      "rewards/rejected": -0.9908422231674194,
       "step": 160
     },
     {
+      "epoch": 0.43478260869565216,
+      "grad_norm": 27.946215188997492,
+      "learning_rate": 3.490199415097892e-08,
+      "logits/chosen": -4.722456932067871,
+      "logits/rejected": -4.946799278259277,
+      "logps/chosen": -246.8963623046875,
+      "logps/rejected": -339.89215087890625,
+      "loss": 0.3253,
+      "rewards/accuracies": 0.90625,
+      "rewards/chosen": 0.14177896082401276,
+      "rewards/margins": 1.3426704406738281,
+      "rewards/rejected": -1.2008916139602661,
       "step": 170
     },
     {
+      "epoch": 0.46035805626598464,
+      "grad_norm": 30.919214925357334,
+      "learning_rate": 3.2810498590513937e-08,
+      "logits/chosen": -4.84631872177124,
+      "logits/rejected": -5.061424255371094,
+      "logps/chosen": -226.3748016357422,
+      "logps/rejected": -312.2584533691406,
+      "loss": 0.3453,
+      "rewards/accuracies": 0.856249988079071,
+      "rewards/chosen": 0.10488839447498322,
+      "rewards/margins": 1.2672706842422485,
+      "rewards/rejected": -1.1623823642730713,
       "step": 180
     },
     {
+      "epoch": 0.4859335038363171,
+      "grad_norm": 35.218898553113945,
+      "learning_rate": 3.065647506074306e-08,
+      "logits/chosen": -4.800053596496582,
+      "logits/rejected": -4.945647716522217,
+      "logps/chosen": -250.15170288085938,
+      "logps/rejected": -353.6054382324219,
+      "loss": 0.3307,
+      "rewards/accuracies": 0.8687499761581421,
+      "rewards/chosen": 0.10741142183542252,
+      "rewards/margins": 1.2918050289154053,
+      "rewards/rejected": -1.1843936443328857,
       "step": 190
     },
     {
+      "epoch": 0.5115089514066496,
+      "grad_norm": 34.27582139337608,
+      "learning_rate": 2.8457167879118325e-08,
+      "logits/chosen": -4.8446269035339355,
+      "logits/rejected": -5.029626846313477,
+      "logps/chosen": -245.35513305664062,
+      "logps/rejected": -340.13250732421875,
+      "loss": 0.3343,
+      "rewards/accuracies": 0.824999988079071,
+      "rewards/chosen": 0.06784123182296753,
+      "rewards/margins": 1.3157011270523071,
+      "rewards/rejected": -1.2478597164154053,
       "step": 200
     },
     {
+      "epoch": 0.5115089514066496,
+      "eval_logits/chosen": -4.865734100341797,
+      "eval_logits/rejected": -5.110110282897949,
+      "eval_logps/chosen": -442.470703125,
+      "eval_logps/rejected": -579.5608520507812,
+      "eval_loss": 0.6981855630874634,
+      "eval_rewards/accuracies": 0.55859375,
+      "eval_rewards/chosen": -0.51996248960495,
+      "eval_rewards/margins": 0.09176936745643616,
+      "eval_rewards/rejected": -0.6117318868637085,
+      "eval_runtime": 98.533,
+      "eval_samples_per_second": 20.298,
+      "eval_steps_per_second": 0.325,
       "step": 200
     },
     {
+      "epoch": 0.5370843989769821,
+      "grad_norm": 27.49657850496402,
+      "learning_rate": 2.6230183887296952e-08,
+      "logits/chosen": -4.9495344161987305,
+      "logits/rejected": -5.170707702636719,
+      "logps/chosen": -253.7008514404297,
+      "logps/rejected": -388.9451904296875,
+      "loss": 0.3045,
+      "rewards/accuracies": 0.9312499761581421,
+      "rewards/chosen": 0.12184244394302368,
+      "rewards/margins": 1.7248340845108032,
+      "rewards/rejected": -1.6029917001724243,
       "step": 210
     },
     {
+      "epoch": 0.5626598465473146,
+      "grad_norm": 31.60560013082535,
+      "learning_rate": 2.3993351497264626e-08,
+      "logits/chosen": -4.796377658843994,
+      "logits/rejected": -5.133594989776611,
+      "logps/chosen": -251.61669921875,
+      "logps/rejected": -385.34405517578125,
+      "loss": 0.3118,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 0.11194615066051483,
+      "rewards/margins": 1.718654990196228,
+      "rewards/rejected": -1.6067088842391968,
       "step": 220
     },
     {
+      "epoch": 0.5882352941176471,
+      "grad_norm": 31.96350965289743,
+      "learning_rate": 2.1764577963648613e-08,
+      "logits/chosen": -4.8748345375061035,
+      "logits/rejected": -5.184638977050781,
+      "logps/chosen": -258.0860900878906,
+      "logps/rejected": -386.3064270019531,
+      "loss": 0.3152,
+      "rewards/accuracies": 0.90625,
+      "rewards/chosen": 0.028830066323280334,
+      "rewards/margins": 1.6547828912734985,
+      "rewards/rejected": -1.6259527206420898,
       "step": 230
     },
     {
+      "epoch": 0.6138107416879796,
+      "grad_norm": 34.58110281626622,
+      "learning_rate": 1.9561706024845818e-08,
+      "logits/chosen": -4.866055488586426,
+      "logits/rejected": -5.124758243560791,
+      "logps/chosen": -271.0610046386719,
+      "logps/rejected": -398.76727294921875,
+      "loss": 0.3015,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 0.017621681094169617,
+      "rewards/margins": 1.7959582805633545,
+      "rewards/rejected": -1.778336763381958,
       "step": 240
     },
     {
+      "epoch": 0.639386189258312,
+      "grad_norm": 34.427123510289576,
+      "learning_rate": 1.740237106064383e-08,
+      "logits/chosen": -5.0298542976379395,
+      "logits/rejected": -5.25708532333374,
+      "logps/chosen": -248.6929168701172,
+      "logps/rejected": -353.6767272949219,
+      "loss": 0.2999,
+      "rewards/accuracies": 0.856249988079071,
+      "rewards/chosen": 8.549987978767604e-05,
+      "rewards/margins": 1.4607607126235962,
+      "rewards/rejected": -1.4606752395629883,
       "step": 250
     },
     {
+      "epoch": 0.6649616368286445,
+      "grad_norm": 36.61434250379136,
+      "learning_rate": 1.530385990987863e-08,
+      "logits/chosen": -4.881124019622803,
+      "logits/rejected": -5.179129600524902,
+      "logps/chosen": -254.1592254638672,
+      "logps/rejected": -414.9335021972656,
+      "loss": 0.2863,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": -0.001106788171455264,
+      "rewards/margins": 1.950743317604065,
+      "rewards/rejected": -1.9518499374389648,
       "step": 260
     },
     {
+      "epoch": 0.690537084398977,
+      "grad_norm": 30.68695587447776,
+      "learning_rate": 1.3282972478382409e-08,
+      "logits/chosen": -5.040741920471191,
+      "logits/rejected": -5.266045570373535,
+      "logps/chosen": -260.43017578125,
+      "logps/rejected": -401.3609924316406,
+      "loss": 0.2997,
+      "rewards/accuracies": 0.918749988079071,
+      "rewards/chosen": 0.006030815653502941,
+      "rewards/margins": 1.8693902492523193,
+      "rewards/rejected": -1.8633596897125244,
       "step": 270
     },
     {
+      "epoch": 0.7161125319693095,
+      "grad_norm": 31.65072997235767,
+      "learning_rate": 1.1355887245137383e-08,
+      "logits/chosen": -4.975480556488037,
+      "logits/rejected": -5.191180229187012,
+      "logps/chosen": -262.6047668457031,
+      "logps/rejected": -436.56243896484375,
+      "loss": 0.2763,
+      "rewards/accuracies": 0.9125000238418579,
+      "rewards/chosen": 0.019021058455109596,
+      "rewards/margins": 1.9552510976791382,
+      "rewards/rejected": -1.9362300634384155,
       "step": 280
     },
     {
+      "epoch": 0.7416879795396419,
+      "grad_norm": 44.14600223942157,
+      "learning_rate": 9.538031743343628e-09,
+      "logits/chosen": -4.857875823974609,
+      "logits/rejected": -5.190948009490967,
+      "logps/chosen": -269.9219055175781,
+      "logps/rejected": -394.4927978515625,
+      "loss": 0.281,
+      "rewards/accuracies": 0.90625,
+      "rewards/chosen": 0.02686493471264839,
+      "rewards/margins": 1.815498948097229,
+      "rewards/rejected": -1.7886340618133545,
       "step": 290
     },
     {
+      "epoch": 0.7672634271099744,
+      "grad_norm": 39.91454735081478,
+      "learning_rate": 7.843959053281662e-09,
+      "logits/chosen": -4.9824910163879395,
+      "logits/rejected": -5.185948371887207,
+      "logps/chosen": -254.4317626953125,
+      "logps/rejected": -409.8275451660156,
+      "loss": 0.2972,
+      "rewards/accuracies": 0.862500011920929,
+      "rewards/chosen": -0.01023593544960022,
+      "rewards/margins": 1.928384780883789,
+      "rewards/rejected": -1.9386205673217773,
       "step": 300
     },
     {
+      "epoch": 0.7672634271099744,
+      "eval_logits/chosen": -5.028459072113037,
+      "eval_logits/rejected": -5.28243350982666,
+      "eval_logps/chosen": -473.4393005371094,
+      "eval_logps/rejected": -613.8079833984375,
+      "eval_loss": 0.7111232280731201,
+      "eval_rewards/accuracies": 0.5625,
+      "eval_rewards/chosen": -0.8296481966972351,
+      "eval_rewards/margins": 0.12455525994300842,
+      "eval_rewards/rejected": -0.9542034864425659,
+      "eval_runtime": 98.5847,
+      "eval_samples_per_second": 20.287,
+      "eval_steps_per_second": 0.325,
       "step": 300
     },
     {
+      "epoch": 0.7928388746803069,
+      "grad_norm": 28.907759477900694,
+      "learning_rate": 6.28723129572247e-09,
+      "logits/chosen": -4.889031887054443,
+      "logits/rejected": -5.217998504638672,
+      "logps/chosen": -259.29278564453125,
+      "logps/rejected": -429.45001220703125,
+      "loss": 0.2863,
+      "rewards/accuracies": 0.925000011920929,
+      "rewards/chosen": 0.01846831850707531,
+      "rewards/margins": 2.0909886360168457,
+      "rewards/rejected": -2.0725202560424805,
       "step": 310
     },
     {
+      "epoch": 0.8184143222506394,
+      "grad_norm": 35.34285789208096,
+      "learning_rate": 4.880311058593617e-09,
+      "logits/chosen": -5.006392002105713,
+      "logits/rejected": -5.3091888427734375,
+      "logps/chosen": -254.5725555419922,
+      "logps/rejected": -404.92706298828125,
+      "loss": 0.2987,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.094082310795784,
+      "rewards/margins": 1.8906733989715576,
+      "rewards/rejected": -1.9847558736801147,
       "step": 320
     },
     {
+      "epoch": 0.8439897698209718,
+      "grad_norm": 31.108205637173466,
+      "learning_rate": 3.6344616260994942e-09,
+      "logits/chosen": -4.935029983520508,
+      "logits/rejected": -5.206645965576172,
+      "logps/chosen": -279.4919128417969,
+      "logps/rejected": -436.8589782714844,
+      "loss": 0.2806,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": -0.11343076080083847,
+      "rewards/margins": 1.9281622171401978,
+      "rewards/rejected": -2.041593074798584,
       "step": 330
     },
     {
+      "epoch": 0.8695652173913043,
+      "grad_norm": 37.630889083981245,
+      "learning_rate": 2.5596568090246547e-09,
+      "logits/chosen": -4.956147193908691,
+      "logits/rejected": -5.322269439697266,
+      "logps/chosen": -274.47479248046875,
+      "logps/rejected": -390.50115966796875,
+      "loss": 0.2837,
+      "rewards/accuracies": 0.8812500238418579,
+      "rewards/chosen": -0.0032872497104108334,
+      "rewards/margins": 1.8339039087295532,
+      "rewards/rejected": -1.8371912240982056,
       "step": 340
     },
     {
+      "epoch": 0.8951406649616368,
+      "grad_norm": 33.50567544254766,
+      "learning_rate": 1.6645010980854079e-09,
+      "logits/chosen": -4.992954254150391,
+      "logits/rejected": -5.131080150604248,
+      "logps/chosen": -267.6521911621094,
+      "logps/rejected": -406.9331359863281,
+      "loss": 0.3087,
+      "rewards/accuracies": 0.90625,
+      "rewards/chosen": -0.12181039899587631,
+      "rewards/margins": 1.66982102394104,
+      "rewards/rejected": -1.7916314601898193,
       "step": 350
     },
     {
+      "epoch": 0.9207161125319693,
+      "grad_norm": 36.41873473091959,
+      "learning_rate": 9.561607795526006e-10,
+      "logits/chosen": -4.958062171936035,
+      "logits/rejected": -5.163455963134766,
+      "logps/chosen": -269.3998107910156,
+      "logps/rejected": -414.72039794921875,
+      "loss": 0.2917,
+      "rewards/accuracies": 0.8812500238418579,
+      "rewards/chosen": -0.07764892280101776,
+      "rewards/margins": 1.7949295043945312,
+      "rewards/rejected": -1.8725783824920654,
       "step": 360
     },
     {
+      "epoch": 0.9462915601023018,
+      "grad_norm": 29.734783261980905,
+      "learning_rate": 4.403065646083809e-10,
+      "logits/chosen": -4.9909467697143555,
+      "logits/rejected": -5.146109104156494,
+      "logps/chosen": -261.77947998046875,
+      "logps/rejected": -422.8021545410156,
+      "loss": 0.292,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": -0.04958271235227585,
+      "rewards/margins": 1.8099679946899414,
+      "rewards/rejected": -1.8595508337020874,
       "step": 370
     },
     {
+      "epoch": 0.9718670076726342,
+      "grad_norm": 37.248731518669295,
+      "learning_rate": 1.2106819172520434e-10,
+      "logits/chosen": -5.114525318145752,
+      "logits/rejected": -5.3700361251831055,
+      "logps/chosen": -265.19122314453125,
+      "logps/rejected": -416.8501892089844,
+      "loss": 0.2878,
+      "rewards/accuracies": 0.887499988079071,
+      "rewards/chosen": -0.0310853011906147,
+      "rewards/margins": 1.9321701526641846,
+      "rewards/rejected": -1.9632552862167358,
       "step": 380
     },
     {
+      "epoch": 0.9974424552429667,
+      "grad_norm": 52.48837519556826,
+      "learning_rate": 1.0013655036916758e-12,
+      "logits/chosen": -5.090936660766602,
+      "logits/rejected": -5.337624549865723,
+      "logps/chosen": -265.92279052734375,
+      "logps/rejected": -421.610595703125,
+      "loss": 0.2755,
+      "rewards/accuracies": 0.8687499761581421,
+      "rewards/chosen": -0.06250263750553131,
+      "rewards/margins": 1.948452353477478,
+      "rewards/rejected": -2.0109550952911377,
       "step": 390
     },
     {
       "epoch": 1.0,
+      "step": 391,
       "total_flos": 0.0,
+      "train_loss": 0.4007269041922391,
+      "train_runtime": 6210.4356,
+      "train_samples_per_second": 8.051,
       "train_steps_per_second": 0.063
     }
   ],
   "logging_steps": 10,
+  "max_steps": 391,
   "num_input_tokens_seen": 0,
   "num_train_epochs": 1,
   "save_steps": 100,

training_args.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cf525e556a4e72ad76dc3263558be495a00b73c02de0b6ea713d4bfeb6a07eb0
-size 6456

 version https://git-lfs.github.com/spec/v1
+oid sha256:0fa7e3e53a21d58800798c272a65a4f1f4bdea1a718fe29f85dc9ce2a41691db
+size 6328