BraylonDash commited on
Commit
4cbd7b3
1 Parent(s): fd2f7d0

Model save

Browse files
README.md ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: peft
4
+ tags:
5
+ - trl
6
+ - dpo
7
+ - generated_from_trainer
8
+ base_model: DUAL-GPO/phi-2-gpo-new-i0
9
+ model-index:
10
+ - name: phi-2-gpo-newSFT-b0.001-15k-i1
11
+ results: []
12
+ ---
13
+
14
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
+ should probably proofread and complete it, then remove this comment. -->
16
+
17
+ # phi-2-gpo-newSFT-b0.001-15k-i1
18
+
19
+ This model is a fine-tuned version of [DUAL-GPO/phi-2-gpo-new-i0](https://huggingface.co/DUAL-GPO/phi-2-gpo-new-i0) on the None dataset.
20
+
21
+ ## Model description
22
+
23
+ More information needed
24
+
25
+ ## Intended uses & limitations
26
+
27
+ More information needed
28
+
29
+ ## Training and evaluation data
30
+
31
+ More information needed
32
+
33
+ ## Training procedure
34
+
35
+ ### Training hyperparameters
36
+
37
+ The following hyperparameters were used during training:
38
+ - learning_rate: 5e-06
39
+ - train_batch_size: 4
40
+ - eval_batch_size: 4
41
+ - seed: 42
42
+ - distributed_type: multi-GPU
43
+ - gradient_accumulation_steps: 4
44
+ - total_train_batch_size: 16
45
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
46
+ - lr_scheduler_type: cosine
47
+ - lr_scheduler_warmup_ratio: 0.1
48
+ - num_epochs: 1
49
+
50
+ ### Training results
51
+
52
+
53
+
54
+ ### Framework versions
55
+
56
+ - PEFT 0.7.1
57
+ - Transformers 4.36.2
58
+ - Pytorch 2.1.2
59
+ - Datasets 2.14.6
60
+ - Tokenizers 0.15.2
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3be8c366aec2b687c60ded55ca416cb823204029c1eac934953e0285d0805b4f
3
  size 167807296
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:04bd7811e39cdf74d4d86fd4c6a4be4cdd840c33cf7ccfcefa091bbccec38ff0
3
  size 167807296
all_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "train_loss": 0.2409534176996449,
4
+ "train_runtime": 8445.1148,
5
+ "train_samples": 61135,
6
+ "train_samples_per_second": 1.776,
7
+ "train_steps_per_second": 0.111
8
+ }
runs/May12_03-16-12_gpu4-119-5/events.out.tfevents.1715447867.gpu4-119-5.1635540.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5bc245304f93c548d3fd6fa64c2d11938730bf5b74e67a3db94e4a5fefd26fab
3
- size 62434
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9115c52186472389cb1a4c5a30f7efa0545b665da1c7bcec376620fc434a1a72
3
+ size 64690
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "train_loss": 0.2409534176996449,
4
+ "train_runtime": 8445.1148,
5
+ "train_samples": 61135,
6
+ "train_samples_per_second": 1.776,
7
+ "train_steps_per_second": 0.111
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,1346 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 0.9994666666666666,
5
+ "eval_steps": 500,
6
+ "global_step": 937,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0,
13
+ "learning_rate": 5.319148936170213e-08,
14
+ "logits/chosen": 0.06159799173474312,
15
+ "logits/rejected": 0.2503473162651062,
16
+ "logps/chosen": -348.4295349121094,
17
+ "logps/rejected": -269.23321533203125,
18
+ "loss": 0.4013,
19
+ "rewards/accuracies": 0.0,
20
+ "rewards/chosen": 0.0,
21
+ "rewards/margins": 0.0,
22
+ "rewards/rejected": 0.0,
23
+ "step": 1
24
+ },
25
+ {
26
+ "epoch": 0.01,
27
+ "learning_rate": 5.319148936170213e-07,
28
+ "logits/chosen": 0.050236668437719345,
29
+ "logits/rejected": 0.23283202946186066,
30
+ "logps/chosen": -357.2872314453125,
31
+ "logps/rejected": -284.6590270996094,
32
+ "loss": 0.3533,
33
+ "rewards/accuracies": 0.3958333432674408,
34
+ "rewards/chosen": -9.200325621350203e-06,
35
+ "rewards/margins": -3.7092313505127095e-06,
36
+ "rewards/rejected": -5.491095635079546e-06,
37
+ "step": 10
38
+ },
39
+ {
40
+ "epoch": 0.02,
41
+ "learning_rate": 1.0638297872340427e-06,
42
+ "logits/chosen": 0.07863714545965195,
43
+ "logits/rejected": 0.17622847855091095,
44
+ "logps/chosen": -305.1004638671875,
45
+ "logps/rejected": -284.56573486328125,
46
+ "loss": 0.3668,
47
+ "rewards/accuracies": 0.46875,
48
+ "rewards/chosen": -3.269618537160568e-05,
49
+ "rewards/margins": -7.148426084313542e-05,
50
+ "rewards/rejected": 3.878808274748735e-05,
51
+ "step": 20
52
+ },
53
+ {
54
+ "epoch": 0.03,
55
+ "learning_rate": 1.595744680851064e-06,
56
+ "logits/chosen": 0.10170497745275497,
57
+ "logits/rejected": 0.2674924433231354,
58
+ "logps/chosen": -318.61309814453125,
59
+ "logps/rejected": -289.11529541015625,
60
+ "loss": 0.3582,
61
+ "rewards/accuracies": 0.44999998807907104,
62
+ "rewards/chosen": -2.7896196570509346e-06,
63
+ "rewards/margins": -0.00010077792830998078,
64
+ "rewards/rejected": 9.798832616070285e-05,
65
+ "step": 30
66
+ },
67
+ {
68
+ "epoch": 0.04,
69
+ "learning_rate": 2.1276595744680853e-06,
70
+ "logits/chosen": 0.09615897387266159,
71
+ "logits/rejected": 0.2696045935153961,
72
+ "logps/chosen": -371.0738220214844,
73
+ "logps/rejected": -285.8421325683594,
74
+ "loss": 0.3338,
75
+ "rewards/accuracies": 0.5249999761581421,
76
+ "rewards/chosen": 4.211660780129023e-05,
77
+ "rewards/margins": 0.0002511069178581238,
78
+ "rewards/rejected": -0.00020899029914289713,
79
+ "step": 40
80
+ },
81
+ {
82
+ "epoch": 0.05,
83
+ "learning_rate": 2.6595744680851065e-06,
84
+ "logits/chosen": 0.1058117151260376,
85
+ "logits/rejected": 0.2665451169013977,
86
+ "logps/chosen": -365.52001953125,
87
+ "logps/rejected": -291.03778076171875,
88
+ "loss": 0.3245,
89
+ "rewards/accuracies": 0.625,
90
+ "rewards/chosen": 0.0004745240439660847,
91
+ "rewards/margins": 0.0008677773294039071,
92
+ "rewards/rejected": -0.00039325328543782234,
93
+ "step": 50
94
+ },
95
+ {
96
+ "epoch": 0.06,
97
+ "learning_rate": 3.191489361702128e-06,
98
+ "logits/chosen": 0.07869451493024826,
99
+ "logits/rejected": 0.2602466642856598,
100
+ "logps/chosen": -347.3607482910156,
101
+ "logps/rejected": -290.67364501953125,
102
+ "loss": 0.3335,
103
+ "rewards/accuracies": 0.731249988079071,
104
+ "rewards/chosen": 0.0010413106065243483,
105
+ "rewards/margins": 0.0015250012511387467,
106
+ "rewards/rejected": -0.00048369058640673757,
107
+ "step": 60
108
+ },
109
+ {
110
+ "epoch": 0.07,
111
+ "learning_rate": 3.723404255319149e-06,
112
+ "logits/chosen": 0.12400275468826294,
113
+ "logits/rejected": 0.26277071237564087,
114
+ "logps/chosen": -336.801025390625,
115
+ "logps/rejected": -295.81439208984375,
116
+ "loss": 0.3541,
117
+ "rewards/accuracies": 0.668749988079071,
118
+ "rewards/chosen": 0.0017572289798408747,
119
+ "rewards/margins": 0.002515274565666914,
120
+ "rewards/rejected": -0.0007580459350720048,
121
+ "step": 70
122
+ },
123
+ {
124
+ "epoch": 0.09,
125
+ "learning_rate": 4.255319148936171e-06,
126
+ "logits/chosen": 0.06505224108695984,
127
+ "logits/rejected": 0.18910573422908783,
128
+ "logps/chosen": -312.43511962890625,
129
+ "logps/rejected": -274.01617431640625,
130
+ "loss": 0.3528,
131
+ "rewards/accuracies": 0.6812499761581421,
132
+ "rewards/chosen": 0.003332150634378195,
133
+ "rewards/margins": 0.004707558546215296,
134
+ "rewards/rejected": -0.0013754077954217792,
135
+ "step": 80
136
+ },
137
+ {
138
+ "epoch": 0.1,
139
+ "learning_rate": 4.787234042553192e-06,
140
+ "logits/chosen": 0.09570185840129852,
141
+ "logits/rejected": 0.292137086391449,
142
+ "logps/chosen": -320.4757080078125,
143
+ "logps/rejected": -282.39453125,
144
+ "loss": 0.3233,
145
+ "rewards/accuracies": 0.706250011920929,
146
+ "rewards/chosen": 0.005724198184907436,
147
+ "rewards/margins": 0.00849069096148014,
148
+ "rewards/rejected": -0.002766493707895279,
149
+ "step": 90
150
+ },
151
+ {
152
+ "epoch": 0.11,
153
+ "learning_rate": 4.999375059004058e-06,
154
+ "logits/chosen": 0.11051477491855621,
155
+ "logits/rejected": 0.2542819380760193,
156
+ "logps/chosen": -341.4449768066406,
157
+ "logps/rejected": -303.7613220214844,
158
+ "loss": 0.2954,
159
+ "rewards/accuracies": 0.7875000238418579,
160
+ "rewards/chosen": 0.006659829057753086,
161
+ "rewards/margins": 0.017101887613534927,
162
+ "rewards/rejected": -0.010442061349749565,
163
+ "step": 100
164
+ },
165
+ {
166
+ "epoch": 0.12,
167
+ "learning_rate": 4.9955571065548795e-06,
168
+ "logits/chosen": 0.09644671529531479,
169
+ "logits/rejected": 0.26117372512817383,
170
+ "logps/chosen": -347.67059326171875,
171
+ "logps/rejected": -339.35205078125,
172
+ "loss": 0.3068,
173
+ "rewards/accuracies": 0.737500011920929,
174
+ "rewards/chosen": -0.005562308244407177,
175
+ "rewards/margins": 0.025120720267295837,
176
+ "rewards/rejected": -0.03068302944302559,
177
+ "step": 110
178
+ },
179
+ {
180
+ "epoch": 0.13,
181
+ "learning_rate": 4.9882736864879e-06,
182
+ "logits/chosen": 0.1643160879611969,
183
+ "logits/rejected": 0.33352339267730713,
184
+ "logps/chosen": -393.2250061035156,
185
+ "logps/rejected": -346.75457763671875,
186
+ "loss": 0.3154,
187
+ "rewards/accuracies": 0.7250000238418579,
188
+ "rewards/chosen": -0.026022305712103844,
189
+ "rewards/margins": 0.04141021519899368,
190
+ "rewards/rejected": -0.06743250787258148,
191
+ "step": 120
192
+ },
193
+ {
194
+ "epoch": 0.14,
195
+ "learning_rate": 4.977534912960124e-06,
196
+ "logits/chosen": 0.14933325350284576,
197
+ "logits/rejected": 0.36655381321907043,
198
+ "logps/chosen": -425.4112243652344,
199
+ "logps/rejected": -428.3099670410156,
200
+ "loss": 0.2788,
201
+ "rewards/accuracies": 0.6875,
202
+ "rewards/chosen": -0.07423840463161469,
203
+ "rewards/margins": 0.06771700084209442,
204
+ "rewards/rejected": -0.1419553905725479,
205
+ "step": 130
206
+ },
207
+ {
208
+ "epoch": 0.15,
209
+ "learning_rate": 4.963355698422092e-06,
210
+ "logits/chosen": 0.20979711413383484,
211
+ "logits/rejected": 0.2947987914085388,
212
+ "logps/chosen": -455.8238830566406,
213
+ "logps/rejected": -478.470703125,
214
+ "loss": 0.2889,
215
+ "rewards/accuracies": 0.6812499761581421,
216
+ "rewards/chosen": -0.11944756656885147,
217
+ "rewards/margins": 0.08143159002065659,
218
+ "rewards/rejected": -0.20087914168834686,
219
+ "step": 140
220
+ },
221
+ {
222
+ "epoch": 0.16,
223
+ "learning_rate": 4.945755732909625e-06,
224
+ "logits/chosen": 0.22968748211860657,
225
+ "logits/rejected": 0.3435000777244568,
226
+ "logps/chosen": -494.1380920410156,
227
+ "logps/rejected": -557.603271484375,
228
+ "loss": 0.2274,
229
+ "rewards/accuracies": 0.78125,
230
+ "rewards/chosen": -0.15503188967704773,
231
+ "rewards/margins": 0.12964466214179993,
232
+ "rewards/rejected": -0.28467655181884766,
233
+ "step": 150
234
+ },
235
+ {
236
+ "epoch": 0.17,
237
+ "learning_rate": 4.924759456701167e-06,
238
+ "logits/chosen": 0.2052285373210907,
239
+ "logits/rejected": 0.3779992461204529,
240
+ "logps/chosen": -555.717529296875,
241
+ "logps/rejected": -618.7412719726562,
242
+ "loss": 0.2945,
243
+ "rewards/accuracies": 0.668749988079071,
244
+ "rewards/chosen": -0.22048434615135193,
245
+ "rewards/margins": 0.12210027873516083,
246
+ "rewards/rejected": -0.34258466958999634,
247
+ "step": 160
248
+ },
249
+ {
250
+ "epoch": 0.18,
251
+ "learning_rate": 4.900396026378671e-06,
252
+ "logits/chosen": 0.19485141336917877,
253
+ "logits/rejected": 0.35038071870803833,
254
+ "logps/chosen": -564.8143310546875,
255
+ "logps/rejected": -614.8881225585938,
256
+ "loss": 0.2459,
257
+ "rewards/accuracies": 0.71875,
258
+ "rewards/chosen": -0.19514423608779907,
259
+ "rewards/margins": 0.13954147696495056,
260
+ "rewards/rejected": -0.33468568325042725,
261
+ "step": 170
262
+ },
263
+ {
264
+ "epoch": 0.19,
265
+ "learning_rate": 4.872699274339169e-06,
266
+ "logits/chosen": 0.26888132095336914,
267
+ "logits/rejected": 0.38210389018058777,
268
+ "logps/chosen": -542.71875,
269
+ "logps/rejected": -610.9818115234375,
270
+ "loss": 0.2668,
271
+ "rewards/accuracies": 0.699999988079071,
272
+ "rewards/chosen": -0.20494337379932404,
273
+ "rewards/margins": 0.13571330904960632,
274
+ "rewards/rejected": -0.34065669775009155,
275
+ "step": 180
276
+ },
277
+ {
278
+ "epoch": 0.2,
279
+ "learning_rate": 4.8417076618132434e-06,
280
+ "logits/chosen": 0.27069562673568726,
281
+ "logits/rejected": 0.3432765603065491,
282
+ "logps/chosen": -552.3577270507812,
283
+ "logps/rejected": -633.7227783203125,
284
+ "loss": 0.2295,
285
+ "rewards/accuracies": 0.7437499761581421,
286
+ "rewards/chosen": -0.20474235713481903,
287
+ "rewards/margins": 0.14836545288562775,
288
+ "rewards/rejected": -0.35310783982276917,
289
+ "step": 190
290
+ },
291
+ {
292
+ "epoch": 0.21,
293
+ "learning_rate": 4.807464225455655e-06,
294
+ "logits/chosen": 0.2959613800048828,
295
+ "logits/rejected": 0.4290657639503479,
296
+ "logps/chosen": -607.9371337890625,
297
+ "logps/rejected": -658.5770263671875,
298
+ "loss": 0.2489,
299
+ "rewards/accuracies": 0.762499988079071,
300
+ "rewards/chosen": -0.22546908259391785,
301
+ "rewards/margins": 0.14855849742889404,
302
+ "rewards/rejected": -0.3740275502204895,
303
+ "step": 200
304
+ },
305
+ {
306
+ "epoch": 0.22,
307
+ "learning_rate": 4.770016517582283e-06,
308
+ "logits/chosen": 0.315132200717926,
309
+ "logits/rejected": 0.43885716795921326,
310
+ "logps/chosen": -636.2320556640625,
311
+ "logps/rejected": -777.607421875,
312
+ "loss": 0.2412,
313
+ "rewards/accuracies": 0.737500011920929,
314
+ "rewards/chosen": -0.28788623213768005,
315
+ "rewards/margins": 0.1952207386493683,
316
+ "rewards/rejected": -0.4831070005893707,
317
+ "step": 210
318
+ },
319
+ {
320
+ "epoch": 0.23,
321
+ "learning_rate": 4.7294165401363616e-06,
322
+ "logits/chosen": 0.27698999643325806,
323
+ "logits/rejected": 0.468730628490448,
324
+ "logps/chosen": -641.8867797851562,
325
+ "logps/rejected": -759.454833984375,
326
+ "loss": 0.2575,
327
+ "rewards/accuracies": 0.737500011920929,
328
+ "rewards/chosen": -0.26833561062812805,
329
+ "rewards/margins": 0.1899726390838623,
330
+ "rewards/rejected": -0.45830821990966797,
331
+ "step": 220
332
+ },
333
+ {
334
+ "epoch": 0.25,
335
+ "learning_rate": 4.68572067247573e-06,
336
+ "logits/chosen": 0.2822573781013489,
337
+ "logits/rejected": 0.42682987451553345,
338
+ "logps/chosen": -578.1446533203125,
339
+ "logps/rejected": -648.0927734375,
340
+ "loss": 0.2443,
341
+ "rewards/accuracies": 0.78125,
342
+ "rewards/chosen": -0.19278240203857422,
343
+ "rewards/margins": 0.18077878654003143,
344
+ "rewards/rejected": -0.37356117367744446,
345
+ "step": 230
346
+ },
347
+ {
348
+ "epoch": 0.26,
349
+ "learning_rate": 4.638989593081364e-06,
350
+ "logits/chosen": 0.3316207230091095,
351
+ "logits/rejected": 0.42460235953330994,
352
+ "logps/chosen": -508.28924560546875,
353
+ "logps/rejected": -627.9178466796875,
354
+ "loss": 0.2394,
355
+ "rewards/accuracies": 0.75,
356
+ "rewards/chosen": -0.18461547791957855,
357
+ "rewards/margins": 0.17850570380687714,
358
+ "rewards/rejected": -0.3631211817264557,
359
+ "step": 240
360
+ },
361
+ {
362
+ "epoch": 0.27,
363
+ "learning_rate": 4.5892881952959015e-06,
364
+ "logits/chosen": 0.34463202953338623,
365
+ "logits/rejected": 0.4213656485080719,
366
+ "logps/chosen": -602.0831909179688,
367
+ "logps/rejected": -723.1624145507812,
368
+ "loss": 0.2229,
369
+ "rewards/accuracies": 0.768750011920929,
370
+ "rewards/chosen": -0.22979848086833954,
371
+ "rewards/margins": 0.1861880123615265,
372
+ "rewards/rejected": -0.41598647832870483,
373
+ "step": 250
374
+ },
375
+ {
376
+ "epoch": 0.28,
377
+ "learning_rate": 4.536685497209182e-06,
378
+ "logits/chosen": 0.2841508984565735,
379
+ "logits/rejected": 0.4096981883049011,
380
+ "logps/chosen": -649.3954467773438,
381
+ "logps/rejected": -714.6544189453125,
382
+ "loss": 0.263,
383
+ "rewards/accuracies": 0.706250011920929,
384
+ "rewards/chosen": -0.26575830578804016,
385
+ "rewards/margins": 0.15293100476264954,
386
+ "rewards/rejected": -0.4186893105506897,
387
+ "step": 260
388
+ },
389
+ {
390
+ "epoch": 0.29,
391
+ "learning_rate": 4.481254545815943e-06,
392
+ "logits/chosen": 0.33302605152130127,
393
+ "logits/rejected": 0.4252694249153137,
394
+ "logps/chosen": -652.48828125,
395
+ "logps/rejected": -734.8269653320312,
396
+ "loss": 0.2375,
397
+ "rewards/accuracies": 0.7875000238418579,
398
+ "rewards/chosen": -0.2703024744987488,
399
+ "rewards/margins": 0.16575393080711365,
400
+ "rewards/rejected": -0.43605637550354004,
401
+ "step": 270
402
+ },
403
+ {
404
+ "epoch": 0.3,
405
+ "learning_rate": 4.42307231557875e-06,
406
+ "logits/chosen": 0.2720186710357666,
407
+ "logits/rejected": 0.4189075827598572,
408
+ "logps/chosen": -606.5438232421875,
409
+ "logps/rejected": -711.4382934570312,
410
+ "loss": 0.234,
411
+ "rewards/accuracies": 0.78125,
412
+ "rewards/chosen": -0.2561277151107788,
413
+ "rewards/margins": 0.1838252693414688,
414
+ "rewards/rejected": -0.4399529993534088,
415
+ "step": 280
416
+ },
417
+ {
418
+ "epoch": 0.31,
419
+ "learning_rate": 4.3622196015370305e-06,
420
+ "logits/chosen": 0.2856540381908417,
421
+ "logits/rejected": 0.46640047430992126,
422
+ "logps/chosen": -581.0446166992188,
423
+ "logps/rejected": -699.5716552734375,
424
+ "loss": 0.2034,
425
+ "rewards/accuracies": 0.7562500238418579,
426
+ "rewards/chosen": -0.22450003027915955,
427
+ "rewards/margins": 0.187502920627594,
428
+ "rewards/rejected": -0.41200295090675354,
429
+ "step": 290
430
+ },
431
+ {
432
+ "epoch": 0.32,
433
+ "learning_rate": 4.298780907110648e-06,
434
+ "logits/chosen": 0.30591830611228943,
435
+ "logits/rejected": 0.4262414574623108,
436
+ "logps/chosen": -547.7359619140625,
437
+ "logps/rejected": -688.0494995117188,
438
+ "loss": 0.2329,
439
+ "rewards/accuracies": 0.7124999761581421,
440
+ "rewards/chosen": -0.2417764961719513,
441
+ "rewards/margins": 0.18638503551483154,
442
+ "rewards/rejected": -0.42816153168678284,
443
+ "step": 300
444
+ },
445
+ {
446
+ "epoch": 0.33,
447
+ "learning_rate": 4.23284432675381e-06,
448
+ "logits/chosen": 0.2821730375289917,
449
+ "logits/rejected": 0.4414825439453125,
450
+ "logps/chosen": -625.08984375,
451
+ "logps/rejected": -736.9857177734375,
452
+ "loss": 0.2392,
453
+ "rewards/accuracies": 0.7250000238418579,
454
+ "rewards/chosen": -0.29913753271102905,
455
+ "rewards/margins": 0.1723407357931137,
456
+ "rewards/rejected": -0.4714782238006592,
457
+ "step": 310
458
+ },
459
+ {
460
+ "epoch": 0.34,
461
+ "learning_rate": 4.164501423622277e-06,
462
+ "logits/chosen": 0.3146980404853821,
463
+ "logits/rejected": 0.5349381566047668,
464
+ "logps/chosen": -667.1527099609375,
465
+ "logps/rejected": -809.9267578125,
466
+ "loss": 0.2347,
467
+ "rewards/accuracies": 0.737500011920929,
468
+ "rewards/chosen": -0.3215456008911133,
469
+ "rewards/margins": 0.2190614491701126,
470
+ "rewards/rejected": -0.5406070947647095,
471
+ "step": 320
472
+ },
473
+ {
474
+ "epoch": 0.35,
475
+ "learning_rate": 4.0938471024237355e-06,
476
+ "logits/chosen": 0.3459940552711487,
477
+ "logits/rejected": 0.5182437300682068,
478
+ "logps/chosen": -632.4589233398438,
479
+ "logps/rejected": -797.7081298828125,
480
+ "loss": 0.1764,
481
+ "rewards/accuracies": 0.862500011920929,
482
+ "rewards/chosen": -0.2630694508552551,
483
+ "rewards/margins": 0.2305426150560379,
484
+ "rewards/rejected": -0.49361199140548706,
485
+ "step": 330
486
+ },
487
+ {
488
+ "epoch": 0.36,
489
+ "learning_rate": 4.020979477627907e-06,
490
+ "logits/chosen": 0.3081711232662201,
491
+ "logits/rejected": 0.3925454020500183,
492
+ "logps/chosen": -592.3121948242188,
493
+ "logps/rejected": -695.0133666992188,
494
+ "loss": 0.2528,
495
+ "rewards/accuracies": 0.706250011920929,
496
+ "rewards/chosen": -0.2648671865463257,
497
+ "rewards/margins": 0.17098167538642883,
498
+ "rewards/rejected": -0.4358488619327545,
499
+ "step": 340
500
+ },
501
+ {
502
+ "epoch": 0.37,
503
+ "learning_rate": 3.9459997372194105e-06,
504
+ "logits/chosen": 0.3076136112213135,
505
+ "logits/rejected": 0.38813453912734985,
506
+ "logps/chosen": -642.5826416015625,
507
+ "logps/rejected": -815.0631103515625,
508
+ "loss": 0.1942,
509
+ "rewards/accuracies": 0.7562500238418579,
510
+ "rewards/chosen": -0.31470799446105957,
511
+ "rewards/margins": 0.20841769874095917,
512
+ "rewards/rejected": -0.5231257677078247,
513
+ "step": 350
514
+ },
515
+ {
516
+ "epoch": 0.38,
517
+ "learning_rate": 3.869012002182573e-06,
518
+ "logits/chosen": 0.2800385057926178,
519
+ "logits/rejected": 0.45763272047042847,
520
+ "logps/chosen": -611.0481567382812,
521
+ "logps/rejected": -790.9743041992188,
522
+ "loss": 0.1966,
523
+ "rewards/accuracies": 0.7562500238418579,
524
+ "rewards/chosen": -0.275074303150177,
525
+ "rewards/margins": 0.22857530415058136,
526
+ "rewards/rejected": -0.5036496520042419,
527
+ "step": 360
528
+ },
529
+ {
530
+ "epoch": 0.39,
531
+ "learning_rate": 3.7901231819133104e-06,
532
+ "logits/chosen": 0.2838382422924042,
533
+ "logits/rejected": 0.4253064692020416,
534
+ "logps/chosen": -663.5225830078125,
535
+ "logps/rejected": -790.7160034179688,
536
+ "loss": 0.2077,
537
+ "rewards/accuracies": 0.737500011920929,
538
+ "rewards/chosen": -0.28215542435646057,
539
+ "rewards/margins": 0.2109585702419281,
540
+ "rewards/rejected": -0.49311399459838867,
541
+ "step": 370
542
+ },
543
+ {
544
+ "epoch": 0.41,
545
+ "learning_rate": 3.709442825758875e-06,
546
+ "logits/chosen": 0.32005783915519714,
547
+ "logits/rejected": 0.42395836114883423,
548
+ "logps/chosen": -589.5636596679688,
549
+ "logps/rejected": -722.5950317382812,
550
+ "loss": 0.2375,
551
+ "rewards/accuracies": 0.7749999761581421,
552
+ "rewards/chosen": -0.24356071650981903,
553
+ "rewards/margins": 0.18706892430782318,
554
+ "rewards/rejected": -0.4306296408176422,
555
+ "step": 380
556
+ },
557
+ {
558
+ "epoch": 0.42,
559
+ "learning_rate": 3.6270829708916113e-06,
560
+ "logits/chosen": 0.318645715713501,
561
+ "logits/rejected": 0.4325946271419525,
562
+ "logps/chosen": -611.2227783203125,
563
+ "logps/rejected": -759.0226440429688,
564
+ "loss": 0.2232,
565
+ "rewards/accuracies": 0.737500011920929,
566
+ "rewards/chosen": -0.25262197852134705,
567
+ "rewards/margins": 0.19418981671333313,
568
+ "rewards/rejected": -0.44681182503700256,
569
+ "step": 390
570
+ },
571
+ {
572
+ "epoch": 0.43,
573
+ "learning_rate": 3.543157986727991e-06,
574
+ "logits/chosen": 0.27690261602401733,
575
+ "logits/rejected": 0.4690285325050354,
576
+ "logps/chosen": -600.1328125,
577
+ "logps/rejected": -722.5906982421875,
578
+ "loss": 0.2403,
579
+ "rewards/accuracies": 0.7124999761581421,
580
+ "rewards/chosen": -0.2763115167617798,
581
+ "rewards/margins": 0.16976875066757202,
582
+ "rewards/rejected": -0.4460802674293518,
583
+ "step": 400
584
+ },
585
+ {
586
+ "epoch": 0.44,
587
+ "learning_rate": 3.4577844161089614e-06,
588
+ "logits/chosen": 0.2978890836238861,
589
+ "logits/rejected": 0.5168454647064209,
590
+ "logps/chosen": -631.8878173828125,
591
+ "logps/rejected": -754.962646484375,
592
+ "loss": 0.2448,
593
+ "rewards/accuracies": 0.737500011920929,
594
+ "rewards/chosen": -0.3049059808254242,
595
+ "rewards/margins": 0.16670657694339752,
596
+ "rewards/rejected": -0.4716125428676605,
597
+ "step": 410
598
+ },
599
+ {
600
+ "epoch": 0.45,
601
+ "learning_rate": 3.3710808134621577e-06,
602
+ "logits/chosen": 0.31090012192726135,
603
+ "logits/rejected": 0.4086151123046875,
604
+ "logps/chosen": -639.5802612304688,
605
+ "logps/rejected": -715.4155883789062,
606
+ "loss": 0.2345,
607
+ "rewards/accuracies": 0.699999988079071,
608
+ "rewards/chosen": -0.29435452818870544,
609
+ "rewards/margins": 0.1740979552268982,
610
+ "rewards/rejected": -0.46845245361328125,
611
+ "step": 420
612
+ },
613
+ {
614
+ "epoch": 0.46,
615
+ "learning_rate": 3.2831675801707126e-06,
616
+ "logits/chosen": 0.29356813430786133,
617
+ "logits/rejected": 0.45901551842689514,
618
+ "logps/chosen": -648.76806640625,
619
+ "logps/rejected": -793.9247436523438,
620
+ "loss": 0.195,
621
+ "rewards/accuracies": 0.7437499761581421,
622
+ "rewards/chosen": -0.30719709396362305,
623
+ "rewards/margins": 0.2049371302127838,
624
+ "rewards/rejected": -0.5121342539787292,
625
+ "step": 430
626
+ },
627
+ {
628
+ "epoch": 0.47,
629
+ "learning_rate": 3.194166797377289e-06,
630
+ "logits/chosen": 0.35314708948135376,
631
+ "logits/rejected": 0.4861513078212738,
632
+ "logps/chosen": -669.8665771484375,
633
+ "logps/rejected": -808.8753051757812,
634
+ "loss": 0.2386,
635
+ "rewards/accuracies": 0.75,
636
+ "rewards/chosen": -0.34487032890319824,
637
+ "rewards/margins": 0.20408621430397034,
638
+ "rewards/rejected": -0.548956573009491,
639
+ "step": 440
640
+ },
641
+ {
642
+ "epoch": 0.48,
643
+ "learning_rate": 3.104202056455501e-06,
644
+ "logits/chosen": 0.3531484007835388,
645
+ "logits/rejected": 0.5174172520637512,
646
+ "logps/chosen": -657.7909545898438,
647
+ "logps/rejected": -806.8094482421875,
648
+ "loss": 0.2039,
649
+ "rewards/accuracies": 0.78125,
650
+ "rewards/chosen": -0.3043491244316101,
651
+ "rewards/margins": 0.21579626202583313,
652
+ "rewards/rejected": -0.5201453566551208,
653
+ "step": 450
654
+ },
655
+ {
656
+ "epoch": 0.49,
657
+ "learning_rate": 3.013398287384144e-06,
658
+ "logits/chosen": 0.29440468549728394,
659
+ "logits/rejected": 0.4629596769809723,
660
+ "logps/chosen": -613.6539916992188,
661
+ "logps/rejected": -718.9657592773438,
662
+ "loss": 0.2514,
663
+ "rewards/accuracies": 0.699999988079071,
664
+ "rewards/chosen": -0.28395384550094604,
665
+ "rewards/margins": 0.17435352504253387,
666
+ "rewards/rejected": -0.4583073556423187,
667
+ "step": 460
668
+ },
669
+ {
670
+ "epoch": 0.5,
671
+ "learning_rate": 2.9218815852625717e-06,
672
+ "logits/chosen": 0.26766303181648254,
673
+ "logits/rejected": 0.4210576117038727,
674
+ "logps/chosen": -611.0421142578125,
675
+ "logps/rejected": -752.10546875,
676
+ "loss": 0.2478,
677
+ "rewards/accuracies": 0.7124999761581421,
678
+ "rewards/chosen": -0.2685486972332001,
679
+ "rewards/margins": 0.20156879723072052,
680
+ "rewards/rejected": -0.4701175093650818,
681
+ "step": 470
682
+ },
683
+ {
684
+ "epoch": 0.51,
685
+ "learning_rate": 2.829779035208113e-06,
686
+ "logits/chosen": 0.3156279921531677,
687
+ "logits/rejected": 0.4234594702720642,
688
+ "logps/chosen": -583.9177856445312,
689
+ "logps/rejected": -695.6286010742188,
690
+ "loss": 0.2687,
691
+ "rewards/accuracies": 0.731249988079071,
692
+ "rewards/chosen": -0.2727667987346649,
693
+ "rewards/margins": 0.16848285496234894,
694
+ "rewards/rejected": -0.44124966859817505,
695
+ "step": 480
696
+ },
697
+ {
698
+ "epoch": 0.52,
699
+ "learning_rate": 2.737218535878705e-06,
700
+ "logits/chosen": 0.2484460324048996,
701
+ "logits/rejected": 0.3799341320991516,
702
+ "logps/chosen": -582.5774536132812,
703
+ "logps/rejected": -692.3199462890625,
704
+ "loss": 0.2314,
705
+ "rewards/accuracies": 0.731249988079071,
706
+ "rewards/chosen": -0.23775920271873474,
707
+ "rewards/margins": 0.1904359757900238,
708
+ "rewards/rejected": -0.42819517850875854,
709
+ "step": 490
710
+ },
711
+ {
712
+ "epoch": 0.53,
713
+ "learning_rate": 2.64432862186579e-06,
714
+ "logits/chosen": 0.31521672010421753,
715
+ "logits/rejected": 0.4720017910003662,
716
+ "logps/chosen": -617.0424194335938,
717
+ "logps/rejected": -763.6742553710938,
718
+ "loss": 0.2106,
719
+ "rewards/accuracies": 0.7875000238418579,
720
+ "rewards/chosen": -0.2587849795818329,
721
+ "rewards/margins": 0.21287491917610168,
722
+ "rewards/rejected": -0.4716598391532898,
723
+ "step": 500
724
+ },
725
+ {
726
+ "epoch": 0.54,
727
+ "learning_rate": 2.551238285204126e-06,
728
+ "logits/chosen": 0.23266558349132538,
729
+ "logits/rejected": 0.4128551483154297,
730
+ "logps/chosen": -645.7574462890625,
731
+ "logps/rejected": -775.3682861328125,
732
+ "loss": 0.2031,
733
+ "rewards/accuracies": 0.737500011920929,
734
+ "rewards/chosen": -0.26859793066978455,
735
+ "rewards/margins": 0.21920350193977356,
736
+ "rewards/rejected": -0.4878014624118805,
737
+ "step": 510
738
+ },
739
+ {
740
+ "epoch": 0.55,
741
+ "learning_rate": 2.4580767962463688e-06,
742
+ "logits/chosen": 0.2809138298034668,
743
+ "logits/rejected": 0.36528998613357544,
744
+ "logps/chosen": -635.5758666992188,
745
+ "logps/rejected": -790.5833129882812,
746
+ "loss": 0.2095,
747
+ "rewards/accuracies": 0.768750011920929,
748
+ "rewards/chosen": -0.29754263162612915,
749
+ "rewards/margins": 0.1984301209449768,
750
+ "rewards/rejected": -0.49597278237342834,
751
+ "step": 520
752
+ },
753
+ {
754
+ "epoch": 0.57,
755
+ "learning_rate": 2.3649735241511546e-06,
756
+ "logits/chosen": 0.27885499596595764,
757
+ "logits/rejected": 0.4202384352684021,
758
+ "logps/chosen": -633.392333984375,
759
+ "logps/rejected": -738.7381591796875,
760
+ "loss": 0.2417,
761
+ "rewards/accuracies": 0.6812499761581421,
762
+ "rewards/chosen": -0.296310693025589,
763
+ "rewards/margins": 0.18409204483032227,
764
+ "rewards/rejected": -0.48040276765823364,
765
+ "step": 530
766
+ },
767
+ {
768
+ "epoch": 0.58,
769
+ "learning_rate": 2.2720577572339914e-06,
770
+ "logits/chosen": 0.2572694420814514,
771
+ "logits/rejected": 0.3829544484615326,
772
+ "logps/chosen": -661.2225341796875,
773
+ "logps/rejected": -775.2647094726562,
774
+ "loss": 0.2215,
775
+ "rewards/accuracies": 0.71875,
776
+ "rewards/chosen": -0.3194531500339508,
777
+ "rewards/margins": 0.1850832998752594,
778
+ "rewards/rejected": -0.5045364499092102,
779
+ "step": 540
780
+ },
781
+ {
782
+ "epoch": 0.59,
783
+ "learning_rate": 2.1794585234303995e-06,
784
+ "logits/chosen": 0.22785460948944092,
785
+ "logits/rejected": 0.39559316635131836,
786
+ "logps/chosen": -616.7025146484375,
787
+ "logps/rejected": -784.1016235351562,
788
+ "loss": 0.2179,
789
+ "rewards/accuracies": 0.7250000238418579,
790
+ "rewards/chosen": -0.28892993927001953,
791
+ "rewards/margins": 0.21685433387756348,
792
+ "rewards/rejected": -0.505784273147583,
793
+ "step": 550
794
+ },
795
+ {
796
+ "epoch": 0.6,
797
+ "learning_rate": 2.0873044111206407e-06,
798
+ "logits/chosen": 0.24686650931835175,
799
+ "logits/rejected": 0.39195549488067627,
800
+ "logps/chosen": -679.55859375,
801
+ "logps/rejected": -784.8621215820312,
802
+ "loss": 0.2101,
803
+ "rewards/accuracies": 0.768750011920929,
804
+ "rewards/chosen": -0.3024412989616394,
805
+ "rewards/margins": 0.1977556049823761,
806
+ "rewards/rejected": -0.5001968741416931,
807
+ "step": 560
808
+ },
809
+ {
810
+ "epoch": 0.61,
811
+ "learning_rate": 1.9957233905648293e-06,
812
+ "logits/chosen": 0.2292439192533493,
813
+ "logits/rejected": 0.4420854151248932,
814
+ "logps/chosen": -635.6842041015625,
815
+ "logps/rejected": -802.7564697265625,
816
+ "loss": 0.2054,
817
+ "rewards/accuracies": 0.7437499761581421,
818
+ "rewards/chosen": -0.2987775206565857,
819
+ "rewards/margins": 0.22222964465618134,
820
+ "rewards/rejected": -0.5210071206092834,
821
+ "step": 570
822
+ },
823
+ {
824
+ "epoch": 0.62,
825
+ "learning_rate": 1.904842636196402e-06,
826
+ "logits/chosen": 0.24736735224723816,
827
+ "logits/rejected": 0.4444567561149597,
828
+ "logps/chosen": -658.0628662109375,
829
+ "logps/rejected": -821.857421875,
830
+ "loss": 0.2104,
831
+ "rewards/accuracies": 0.768750011920929,
832
+ "rewards/chosen": -0.3125050663948059,
833
+ "rewards/margins": 0.21667668223381042,
834
+ "rewards/rejected": -0.5291817784309387,
835
+ "step": 580
836
+ },
837
+ {
838
+ "epoch": 0.63,
839
+ "learning_rate": 1.814788350020726e-06,
840
+ "logits/chosen": 0.18406185507774353,
841
+ "logits/rejected": 0.3239056468009949,
842
+ "logps/chosen": -618.2083129882812,
843
+ "logps/rejected": -770.7822875976562,
844
+ "loss": 0.2304,
845
+ "rewards/accuracies": 0.7124999761581421,
846
+ "rewards/chosen": -0.2990264892578125,
847
+ "rewards/margins": 0.19497379660606384,
848
+ "rewards/rejected": -0.49400028586387634,
849
+ "step": 590
850
+ },
851
+ {
852
+ "epoch": 0.64,
853
+ "learning_rate": 1.725685586364051e-06,
854
+ "logits/chosen": 0.23465320467948914,
855
+ "logits/rejected": 0.41701340675354004,
856
+ "logps/chosen": -603.99609375,
857
+ "logps/rejected": -717.4134521484375,
858
+ "loss": 0.2277,
859
+ "rewards/accuracies": 0.6937500238418579,
860
+ "rewards/chosen": -0.274019718170166,
861
+ "rewards/margins": 0.18845617771148682,
862
+ "rewards/rejected": -0.4624759256839752,
863
+ "step": 600
864
+ },
865
+ {
866
+ "epoch": 0.65,
867
+ "learning_rate": 1.6376580782162172e-06,
868
+ "logits/chosen": 0.2198990285396576,
869
+ "logits/rejected": 0.3981742560863495,
870
+ "logps/chosen": -660.04931640625,
871
+ "logps/rejected": -776.05322265625,
872
+ "loss": 0.2166,
873
+ "rewards/accuracies": 0.75,
874
+ "rewards/chosen": -0.275198370218277,
875
+ "rewards/margins": 0.21421167254447937,
876
+ "rewards/rejected": -0.4894101023674011,
877
+ "step": 610
878
+ },
879
+ {
880
+ "epoch": 0.66,
881
+ "learning_rate": 1.550828065408227e-06,
882
+ "logits/chosen": 0.21209344267845154,
883
+ "logits/rejected": 0.35057374835014343,
884
+ "logps/chosen": -608.8699951171875,
885
+ "logps/rejected": -748.3526611328125,
886
+ "loss": 0.2356,
887
+ "rewards/accuracies": 0.706250011920929,
888
+ "rewards/chosen": -0.30509600043296814,
889
+ "rewards/margins": 0.183022603392601,
890
+ "rewards/rejected": -0.48811858892440796,
891
+ "step": 620
892
+ },
893
+ {
894
+ "epoch": 0.67,
895
+ "learning_rate": 1.4653161248633053e-06,
896
+ "logits/chosen": 0.1554277241230011,
897
+ "logits/rejected": 0.38564786314964294,
898
+ "logps/chosen": -655.8323974609375,
899
+ "logps/rejected": -804.4343872070312,
900
+ "loss": 0.2157,
901
+ "rewards/accuracies": 0.762499988079071,
902
+ "rewards/chosen": -0.29179584980010986,
903
+ "rewards/margins": 0.22734172642230988,
904
+ "rewards/rejected": -0.5191375613212585,
905
+ "step": 630
906
+ },
907
+ {
908
+ "epoch": 0.68,
909
+ "learning_rate": 1.381241003157162e-06,
910
+ "logits/chosen": 0.21835489571094513,
911
+ "logits/rejected": 0.365634948015213,
912
+ "logps/chosen": -661.9124145507812,
913
+ "logps/rejected": -783.0175170898438,
914
+ "loss": 0.1961,
915
+ "rewards/accuracies": 0.7437499761581421,
916
+ "rewards/chosen": -0.3171085715293884,
917
+ "rewards/margins": 0.20071883499622345,
918
+ "rewards/rejected": -0.5178273916244507,
919
+ "step": 640
920
+ },
921
+ {
922
+ "epoch": 0.69,
923
+ "learning_rate": 1.298719451619979e-06,
924
+ "logits/chosen": 0.23657183349132538,
925
+ "logits/rejected": 0.3743918836116791,
926
+ "logps/chosen": -680.4862670898438,
927
+ "logps/rejected": -835.0526123046875,
928
+ "loss": 0.2212,
929
+ "rewards/accuracies": 0.762499988079071,
930
+ "rewards/chosen": -0.3263990879058838,
931
+ "rewards/margins": 0.22416400909423828,
932
+ "rewards/rejected": -0.5505631566047668,
933
+ "step": 650
934
+ },
935
+ {
936
+ "epoch": 0.7,
937
+ "learning_rate": 1.2178660642091036e-06,
938
+ "logits/chosen": 0.2182171642780304,
939
+ "logits/rejected": 0.32706716656684875,
940
+ "logps/chosen": -675.8248901367188,
941
+ "logps/rejected": -836.1906127929688,
942
+ "loss": 0.2204,
943
+ "rewards/accuracies": 0.75,
944
+ "rewards/chosen": -0.3406047821044922,
945
+ "rewards/margins": 0.20144791901111603,
946
+ "rewards/rejected": -0.5420526266098022,
947
+ "step": 660
948
+ },
949
+ {
950
+ "epoch": 0.71,
951
+ "learning_rate": 1.1387931183775821e-06,
952
+ "logits/chosen": 0.21965551376342773,
953
+ "logits/rejected": 0.377861887216568,
954
+ "logps/chosen": -645.6212158203125,
955
+ "logps/rejected": -850.5048828125,
956
+ "loss": 0.2011,
957
+ "rewards/accuracies": 0.7562500238418579,
958
+ "rewards/chosen": -0.32531431317329407,
959
+ "rewards/margins": 0.23981444537639618,
960
+ "rewards/rejected": -0.565128743648529,
961
+ "step": 670
962
+ },
963
+ {
964
+ "epoch": 0.73,
965
+ "learning_rate": 1.061610419159532e-06,
966
+ "logits/chosen": 0.23912313580513,
967
+ "logits/rejected": 0.33457493782043457,
968
+ "logps/chosen": -652.5158081054688,
969
+ "logps/rejected": -804.5699462890625,
970
+ "loss": 0.2,
971
+ "rewards/accuracies": 0.71875,
972
+ "rewards/chosen": -0.3123088479042053,
973
+ "rewards/margins": 0.22432692348957062,
974
+ "rewards/rejected": -0.5366358160972595,
975
+ "step": 680
976
+ },
977
+ {
978
+ "epoch": 0.74,
979
+ "learning_rate": 9.864251466888364e-07,
980
+ "logits/chosen": 0.22238676249980927,
981
+ "logits/rejected": 0.4501045346260071,
982
+ "logps/chosen": -702.1354370117188,
983
+ "logps/rejected": -854.2493896484375,
984
+ "loss": 0.2429,
985
+ "rewards/accuracies": 0.737500011920929,
986
+ "rewards/chosen": -0.3496148884296417,
987
+ "rewards/margins": 0.21850232779979706,
988
+ "rewards/rejected": -0.5681171417236328,
989
+ "step": 690
990
+ },
991
+ {
992
+ "epoch": 0.75,
993
+ "learning_rate": 9.133417073629288e-07,
994
+ "logits/chosen": 0.21312983334064484,
995
+ "logits/rejected": 0.3885112702846527,
996
+ "logps/chosen": -668.33447265625,
997
+ "logps/rejected": -807.5436401367188,
998
+ "loss": 0.236,
999
+ "rewards/accuracies": 0.6875,
1000
+ "rewards/chosen": -0.3556881546974182,
1001
+ "rewards/margins": 0.1771022081375122,
1002
+ "rewards/rejected": -0.5327903628349304,
1003
+ "step": 700
1004
+ },
1005
+ {
1006
+ "epoch": 0.76,
1007
+ "learning_rate": 8.424615888583332e-07,
1008
+ "logits/chosen": 0.17015187442302704,
1009
+ "logits/rejected": 0.3787848949432373,
1010
+ "logps/chosen": -694.0350341796875,
1011
+ "logps/rejected": -877.3869018554688,
1012
+ "loss": 0.1718,
1013
+ "rewards/accuracies": 0.800000011920929,
1014
+ "rewards/chosen": -0.32832252979278564,
1015
+ "rewards/margins": 0.2480977475643158,
1016
+ "rewards/rejected": -0.5764203071594238,
1017
+ "step": 710
1018
+ },
1019
+ {
1020
+ "epoch": 0.77,
1021
+ "learning_rate": 7.738832191993092e-07,
1022
+ "logits/chosen": 0.18861600756645203,
1023
+ "logits/rejected": 0.3507705330848694,
1024
+ "logps/chosen": -677.43701171875,
1025
+ "logps/rejected": -824.4730224609375,
1026
+ "loss": 0.1952,
1027
+ "rewards/accuracies": 0.8125,
1028
+ "rewards/chosen": -0.3222949504852295,
1029
+ "rewards/margins": 0.23219437897205353,
1030
+ "rewards/rejected": -0.5544893145561218,
1031
+ "step": 720
1032
+ },
1033
+ {
1034
+ "epoch": 0.78,
1035
+ "learning_rate": 7.077018300752917e-07,
1036
+ "logits/chosen": 0.22210793197155,
1037
+ "logits/rejected": 0.3462095856666565,
1038
+ "logps/chosen": -642.8484497070312,
1039
+ "logps/rejected": -801.2487182617188,
1040
+ "loss": 0.2144,
1041
+ "rewards/accuracies": 0.7562500238418579,
1042
+ "rewards/chosen": -0.2975847125053406,
1043
+ "rewards/margins": 0.23604163527488708,
1044
+ "rewards/rejected": -0.53362637758255,
1045
+ "step": 730
1046
+ },
1047
+ {
1048
+ "epoch": 0.79,
1049
+ "learning_rate": 6.440093245969342e-07,
1050
+ "logits/chosen": 0.18037883937358856,
1051
+ "logits/rejected": 0.32701772451400757,
1052
+ "logps/chosen": -619.3107299804688,
1053
+ "logps/rejected": -796.5120849609375,
1054
+ "loss": 0.2121,
1055
+ "rewards/accuracies": 0.699999988079071,
1056
+ "rewards/chosen": -0.2894679307937622,
1057
+ "rewards/margins": 0.22074396908283234,
1058
+ "rewards/rejected": -0.5102118849754333,
1059
+ "step": 740
1060
+ },
1061
+ {
1062
+ "epoch": 0.8,
1063
+ "learning_rate": 5.828941496744075e-07,
1064
+ "logits/chosen": 0.21488244831562042,
1065
+ "logits/rejected": 0.3539666533470154,
1066
+ "logps/chosen": -646.1163330078125,
1067
+ "logps/rejected": -821.9992065429688,
1068
+ "loss": 0.1931,
1069
+ "rewards/accuracies": 0.78125,
1070
+ "rewards/chosen": -0.30917179584503174,
1071
+ "rewards/margins": 0.22842054069042206,
1072
+ "rewards/rejected": -0.537592351436615,
1073
+ "step": 750
1074
+ },
1075
+ {
1076
+ "epoch": 0.81,
1077
+ "learning_rate": 5.244411731951671e-07,
1078
+ "logits/chosen": 0.2123112678527832,
1079
+ "logits/rejected": 0.38105306029319763,
1080
+ "logps/chosen": -718.8529663085938,
1081
+ "logps/rejected": -841.3015747070312,
1082
+ "loss": 0.2265,
1083
+ "rewards/accuracies": 0.7875000238418579,
1084
+ "rewards/chosen": -0.3423205316066742,
1085
+ "rewards/margins": 0.21526296436786652,
1086
+ "rewards/rejected": -0.5575835108757019,
1087
+ "step": 760
1088
+ },
1089
+ {
1090
+ "epoch": 0.82,
1091
+ "learning_rate": 4.6873156617173594e-07,
1092
+ "logits/chosen": 0.2072085589170456,
1093
+ "logits/rejected": 0.33443737030029297,
1094
+ "logps/chosen": -687.7586669921875,
1095
+ "logps/rejected": -815.7947998046875,
1096
+ "loss": 0.2148,
1097
+ "rewards/accuracies": 0.731249988079071,
1098
+ "rewards/chosen": -0.3153381943702698,
1099
+ "rewards/margins": 0.20979492366313934,
1100
+ "rewards/rejected": -0.5251331329345703,
1101
+ "step": 770
1102
+ },
1103
+ {
1104
+ "epoch": 0.83,
1105
+ "learning_rate": 4.1584269002318653e-07,
1106
+ "logits/chosen": 0.22315290570259094,
1107
+ "logits/rejected": 0.39601173996925354,
1108
+ "logps/chosen": -641.2395629882812,
1109
+ "logps/rejected": -768.5919799804688,
1110
+ "loss": 0.2332,
1111
+ "rewards/accuracies": 0.762499988079071,
1112
+ "rewards/chosen": -0.2931545674800873,
1113
+ "rewards/margins": 0.20355018973350525,
1114
+ "rewards/rejected": -0.49670475721359253,
1115
+ "step": 780
1116
+ },
1117
+ {
1118
+ "epoch": 0.84,
1119
+ "learning_rate": 3.658479891468258e-07,
1120
+ "logits/chosen": 0.23425737023353577,
1121
+ "logits/rejected": 0.3700588047504425,
1122
+ "logps/chosen": -647.1505737304688,
1123
+ "logps/rejected": -796.8721313476562,
1124
+ "loss": 0.2232,
1125
+ "rewards/accuracies": 0.737500011920929,
1126
+ "rewards/chosen": -0.3021949827671051,
1127
+ "rewards/margins": 0.2227589339017868,
1128
+ "rewards/rejected": -0.5249539017677307,
1129
+ "step": 790
1130
+ },
1131
+ {
1132
+ "epoch": 0.85,
1133
+ "learning_rate": 3.18816888929272e-07,
1134
+ "logits/chosen": 0.21941573917865753,
1135
+ "logits/rejected": 0.38576945662498474,
1136
+ "logps/chosen": -681.2139282226562,
1137
+ "logps/rejected": -785.1143188476562,
1138
+ "loss": 0.2456,
1139
+ "rewards/accuracies": 0.65625,
1140
+ "rewards/chosen": -0.3359334468841553,
1141
+ "rewards/margins": 0.15878985822200775,
1142
+ "rewards/rejected": -0.4947233200073242,
1143
+ "step": 800
1144
+ },
1145
+ {
1146
+ "epoch": 0.86,
1147
+ "learning_rate": 2.748146993385484e-07,
1148
+ "logits/chosen": 0.21103620529174805,
1149
+ "logits/rejected": 0.37812119722366333,
1150
+ "logps/chosen": -675.7420654296875,
1151
+ "logps/rejected": -835.7942504882812,
1152
+ "loss": 0.2342,
1153
+ "rewards/accuracies": 0.737500011920929,
1154
+ "rewards/chosen": -0.3283590078353882,
1155
+ "rewards/margins": 0.1955871880054474,
1156
+ "rewards/rejected": -0.5239461660385132,
1157
+ "step": 810
1158
+ },
1159
+ {
1160
+ "epoch": 0.87,
1161
+ "learning_rate": 2.3390252423108077e-07,
1162
+ "logits/chosen": 0.18820640444755554,
1163
+ "logits/rejected": 0.3453311324119568,
1164
+ "logps/chosen": -699.3671264648438,
1165
+ "logps/rejected": -808.0126342773438,
1166
+ "loss": 0.2185,
1167
+ "rewards/accuracies": 0.7437499761581421,
1168
+ "rewards/chosen": -0.32744988799095154,
1169
+ "rewards/margins": 0.19639787077903748,
1170
+ "rewards/rejected": -0.523847758769989,
1171
+ "step": 820
1172
+ },
1173
+ {
1174
+ "epoch": 0.89,
1175
+ "learning_rate": 1.961371764995243e-07,
1176
+ "logits/chosen": 0.1902572214603424,
1177
+ "logits/rejected": 0.3874988257884979,
1178
+ "logps/chosen": -704.2755737304688,
1179
+ "logps/rejected": -798.5086669921875,
1180
+ "loss": 0.2272,
1181
+ "rewards/accuracies": 0.7250000238418579,
1182
+ "rewards/chosen": -0.3353065848350525,
1183
+ "rewards/margins": 0.1846623420715332,
1184
+ "rewards/rejected": -0.5199689269065857,
1185
+ "step": 830
1186
+ },
1187
+ {
1188
+ "epoch": 0.9,
1189
+ "learning_rate": 1.61571099179261e-07,
1190
+ "logits/chosen": 0.2346924990415573,
1191
+ "logits/rejected": 0.35877925157546997,
1192
+ "logps/chosen": -631.2687377929688,
1193
+ "logps/rejected": -800.9693603515625,
1194
+ "loss": 0.2343,
1195
+ "rewards/accuracies": 0.7562500238418579,
1196
+ "rewards/chosen": -0.31699609756469727,
1197
+ "rewards/margins": 0.20049229264259338,
1198
+ "rewards/rejected": -0.517488420009613,
1199
+ "step": 840
1200
+ },
1201
+ {
1202
+ "epoch": 0.91,
1203
+ "learning_rate": 1.3025229262312367e-07,
1204
+ "logits/chosen": 0.22632627189159393,
1205
+ "logits/rejected": 0.4042627811431885,
1206
+ "logps/chosen": -647.9192504882812,
1207
+ "logps/rejected": -805.1476440429688,
1208
+ "loss": 0.1767,
1209
+ "rewards/accuracies": 0.7875000238418579,
1210
+ "rewards/chosen": -0.29563456773757935,
1211
+ "rewards/margins": 0.23449036478996277,
1212
+ "rewards/rejected": -0.5301249623298645,
1213
+ "step": 850
1214
+ },
1215
+ {
1216
+ "epoch": 0.92,
1217
+ "learning_rate": 1.0222424784546853e-07,
1218
+ "logits/chosen": 0.18376657366752625,
1219
+ "logits/rejected": 0.2582804560661316,
1220
+ "logps/chosen": -614.8178100585938,
1221
+ "logps/rejected": -754.7501220703125,
1222
+ "loss": 0.2084,
1223
+ "rewards/accuracies": 0.699999988079071,
1224
+ "rewards/chosen": -0.27827420830726624,
1225
+ "rewards/margins": 0.21241740882396698,
1226
+ "rewards/rejected": -0.49069157242774963,
1227
+ "step": 860
1228
+ },
1229
+ {
1230
+ "epoch": 0.93,
1231
+ "learning_rate": 7.752588612816553e-08,
1232
+ "logits/chosen": 0.2309151142835617,
1233
+ "logits/rejected": 0.32766053080558777,
1234
+ "logps/chosen": -618.4566650390625,
1235
+ "logps/rejected": -769.9896850585938,
1236
+ "loss": 0.2188,
1237
+ "rewards/accuracies": 0.699999988079071,
1238
+ "rewards/chosen": -0.2959611415863037,
1239
+ "rewards/margins": 0.19887039065361023,
1240
+ "rewards/rejected": -0.49483156204223633,
1241
+ "step": 870
1242
+ },
1243
+ {
1244
+ "epoch": 0.94,
1245
+ "learning_rate": 5.619150497236991e-08,
1246
+ "logits/chosen": 0.24140353500843048,
1247
+ "logits/rejected": 0.29053181409835815,
1248
+ "logps/chosen": -647.7692260742188,
1249
+ "logps/rejected": -806.162109375,
1250
+ "loss": 0.2179,
1251
+ "rewards/accuracies": 0.75,
1252
+ "rewards/chosen": -0.31588608026504517,
1253
+ "rewards/margins": 0.21005694568157196,
1254
+ "rewards/rejected": -0.5259430408477783,
1255
+ "step": 880
1256
+ },
1257
+ {
1258
+ "epoch": 0.95,
1259
+ "learning_rate": 3.825073047112743e-08,
1260
+ "logits/chosen": 0.21158334612846375,
1261
+ "logits/rejected": 0.31529536843299866,
1262
+ "logps/chosen": -656.6387939453125,
1263
+ "logps/rejected": -829.8762817382812,
1264
+ "loss": 0.2066,
1265
+ "rewards/accuracies": 0.800000011920929,
1266
+ "rewards/chosen": -0.31754207611083984,
1267
+ "rewards/margins": 0.2269022911787033,
1268
+ "rewards/rejected": -0.5444443821907043,
1269
+ "step": 890
1270
+ },
1271
+ {
1272
+ "epoch": 0.96,
1273
+ "learning_rate": 2.372847616895685e-08,
1274
+ "logits/chosen": 0.18535788357257843,
1275
+ "logits/rejected": 0.3380703330039978,
1276
+ "logps/chosen": -669.4324340820312,
1277
+ "logps/rejected": -809.5096435546875,
1278
+ "loss": 0.2213,
1279
+ "rewards/accuracies": 0.737500011920929,
1280
+ "rewards/chosen": -0.33157414197921753,
1281
+ "rewards/margins": 0.20540854334831238,
1282
+ "rewards/rejected": -0.5369827151298523,
1283
+ "step": 900
1284
+ },
1285
+ {
1286
+ "epoch": 0.97,
1287
+ "learning_rate": 1.264490846553279e-08,
1288
+ "logits/chosen": 0.21935626864433289,
1289
+ "logits/rejected": 0.3299446702003479,
1290
+ "logps/chosen": -643.43310546875,
1291
+ "logps/rejected": -753.556884765625,
1292
+ "loss": 0.2527,
1293
+ "rewards/accuracies": 0.699999988079071,
1294
+ "rewards/chosen": -0.3096220791339874,
1295
+ "rewards/margins": 0.17697684466838837,
1296
+ "rewards/rejected": -0.4865988790988922,
1297
+ "step": 910
1298
+ },
1299
+ {
1300
+ "epoch": 0.98,
1301
+ "learning_rate": 5.015418611516165e-09,
1302
+ "logits/chosen": 0.20347245037555695,
1303
+ "logits/rejected": 0.37708914279937744,
1304
+ "logps/chosen": -650.4703979492188,
1305
+ "logps/rejected": -769.6436767578125,
1306
+ "loss": 0.2711,
1307
+ "rewards/accuracies": 0.6625000238418579,
1308
+ "rewards/chosen": -0.3225279450416565,
1309
+ "rewards/margins": 0.16174957156181335,
1310
+ "rewards/rejected": -0.48427754640579224,
1311
+ "step": 920
1312
+ },
1313
+ {
1314
+ "epoch": 0.99,
1315
+ "learning_rate": 8.506013354186993e-10,
1316
+ "logits/chosen": 0.22470831871032715,
1317
+ "logits/rejected": 0.3196450173854828,
1318
+ "logps/chosen": -647.2747192382812,
1319
+ "logps/rejected": -821.5018310546875,
1320
+ "loss": 0.2047,
1321
+ "rewards/accuracies": 0.7562500238418579,
1322
+ "rewards/chosen": -0.3270157277584076,
1323
+ "rewards/margins": 0.21042361855506897,
1324
+ "rewards/rejected": -0.5374393463134766,
1325
+ "step": 930
1326
+ },
1327
+ {
1328
+ "epoch": 1.0,
1329
+ "step": 937,
1330
+ "total_flos": 0.0,
1331
+ "train_loss": 0.2409534176996449,
1332
+ "train_runtime": 8445.1148,
1333
+ "train_samples_per_second": 1.776,
1334
+ "train_steps_per_second": 0.111
1335
+ }
1336
+ ],
1337
+ "logging_steps": 10,
1338
+ "max_steps": 937,
1339
+ "num_input_tokens_seen": 0,
1340
+ "num_train_epochs": 1,
1341
+ "save_steps": 100,
1342
+ "total_flos": 0.0,
1343
+ "train_batch_size": 4,
1344
+ "trial_name": null,
1345
+ "trial_params": null
1346
+ }