Text Generation
Transformers
PyTorch
English
llama
finance
llms
text-generation-inference
Weiguang Han commited on
Commit
09b4141
1 Parent(s): 2862848

update tokenizer

Browse files
README.md CHANGED
@@ -1,3 +1,8 @@
1
  ---
2
  license: mit
3
  ---
 
 
 
 
 
 
1
  ---
2
  license: mit
3
  ---
4
+ ```Python
5
+ from transformers import LlamaTokenizer, LlamaForCausalLM
6
+ LlamaTokenizer.from_pretrained('ChanceFocus/finma-7b-nlp')
7
+ LlamaForCausalLM.from_pretrained('ChanceFocus/finma-7b-nlp', device_map='auto')
8
+ ```
global_step3639/zero_pp_rank_0_mp_rank_00_optim_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:03df8e22a4c413dc584f686b04542fe2a60f136c24ed945e29d79d3017704504
3
- size 10107627001
 
 
 
 
global_step3639/zero_pp_rank_1_mp_rank_00_model_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:9d94cf7316f596e4de4c49cf79a5ac7642c5b220d7664f54ad0beace3fe97fc8
3
- size 168022
 
 
 
 
global_step3639/zero_pp_rank_1_mp_rank_00_optim_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:fcc119a3938eb92d256d95c08d408c8e8556fcf10a8f771aa567ac42b56c4a9a
3
- size 10107627001
 
 
 
 
global_step3639/zero_pp_rank_2_mp_rank_00_model_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:fbfc3a858a6c1d6da9c68767928220efa9f7806cf357eb56d277a55650d179ba
3
- size 168022
 
 
 
 
global_step3639/zero_pp_rank_2_mp_rank_00_optim_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:b010db939a59b1eb80d553539d581589c9dd3066b1799410f75774bb877751a1
3
- size 10107627001
 
 
 
 
global_step3639/zero_pp_rank_3_mp_rank_00_model_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:e7144ab90a39d68984bd0afc781c1aed3971815f744acf46bb76e23ca1c8e6e0
3
- size 168022
 
 
 
 
global_step3639/zero_pp_rank_3_mp_rank_00_optim_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:0e3826a7991563a7a3fd5406075a47695c48840c3119ad95ea2a4d1b43b9dd0f
3
- size 10107627001
 
 
 
 
global_step3639/zero_pp_rank_4_mp_rank_00_model_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:b483ace6acbaf9afae274369ea879ebbff802b037ca6f91fffaed30900b0960b
3
- size 168022
 
 
 
 
global_step3639/zero_pp_rank_4_mp_rank_00_optim_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:c73df25ad3243114efaee856ac7334a6bf66413c6da50141d4f7dc15d5ad148b
3
- size 10107627001
 
 
 
 
global_step3639/zero_pp_rank_5_mp_rank_00_model_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:b8d548c30c216cd78db737ba16d8df715a43bae2477163f11007a184fdf9d0eb
3
- size 168022
 
 
 
 
global_step3639/zero_pp_rank_5_mp_rank_00_optim_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:bce9cdebe47cd06b7065fd17c8b8ba6b79cefffc41471d94a3e11c2f125441d4
3
- size 10107627001
 
 
 
 
global_step3639/zero_pp_rank_6_mp_rank_00_model_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:724292452affe5ff8d98db96ebb9f356bd72324f32e2c281e23b608ddc3f245f
3
- size 168022
 
 
 
 
global_step3639/zero_pp_rank_6_mp_rank_00_optim_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:0ed58c2e3cf103ce762ac8dba241d0946025d71b96874710e43af964df801f7a
3
- size 10107627001
 
 
 
 
global_step3639/zero_pp_rank_7_mp_rank_00_model_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:dd7630f212570e2624b64ae629c046edc7747508e61846faff7da6c3c22dd3ce
3
- size 168022
 
 
 
 
global_step3639/zero_pp_rank_7_mp_rank_00_optim_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:ead37ef4a3a2c87d53960d04995431d06eae49d3fa01838b42dd2954f5217320
3
- size 10107627001
 
 
 
 
latest DELETED
@@ -1 +0,0 @@
1
- global_step3639
 
 
rng_state_0.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:5539dd7b259c1dd70b24a442cd0db654de172851dc9ad592c00bd0873776a675
3
- size 21687
 
 
 
 
rng_state_1.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:ae401ae645379434caffc0185c738039f26685db09ba0aba7a187a8ff97810a9
3
- size 21687
 
 
 
 
rng_state_2.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:c89a43885840bc17c1afdbf25def2d1f36021795ec85893c0d9b1fe1c5b8540d
3
- size 21687
 
 
 
 
rng_state_3.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:47edb967b5d8d4551ccb3bda2bf617d308d0981d37333b4a8839054f68577695
3
- size 21687
 
 
 
 
rng_state_4.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:fef2af776d8df77a340254ef61a2b8d0fcf40ee1ddfaeca625988e0165237db3
3
- size 21687
 
 
 
 
rng_state_5.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:b1d3372a72c3e55207954229dad35b30e29127caffc546802c219fbbecc37247
3
- size 21687
 
 
 
 
rng_state_6.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:5903aacbc3201df025352bcb75c398360d0239d5813369122cc0785e94aa68dc
3
- size 21687
 
 
 
 
rng_state_7.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:b0102f09cbe2f6f8f4be258e9801fe1495308691ad96ede2d8c503f2d48b5d4e
3
- size 21687
 
 
 
 
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "",
11
+ "lstrip": false,
12
+ "normalized": true,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "unk_token": {
17
+ "content": "",
18
+ "lstrip": false,
19
+ "normalized": true,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ }
23
+ }
global_step3639/zero_pp_rank_0_mp_rank_00_model_states.pt → tokenizer.model RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:00b03898cdd9f6dfd5a25172d6b1a59216396c5cadd8df564b402f191a02e312
3
- size 168022
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
3
+ size 499723
tokenizer_config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "bos_token": {
5
+ "__type": "AddedToken",
6
+ "content": "",
7
+ "lstrip": false,
8
+ "normalized": true,
9
+ "rstrip": false,
10
+ "single_word": false
11
+ },
12
+ "clean_up_tokenization_spaces": false,
13
+ "eos_token": {
14
+ "__type": "AddedToken",
15
+ "content": "",
16
+ "lstrip": false,
17
+ "normalized": true,
18
+ "rstrip": false,
19
+ "single_word": false
20
+ },
21
+ "model_max_length": 1000000000000000019884624838656,
22
+ "pad_token": null,
23
+ "sp_model_kwargs": {},
24
+ "tokenizer_class": "LlamaTokenizer",
25
+ "unk_token": {
26
+ "__type": "AddedToken",
27
+ "content": "",
28
+ "lstrip": false,
29
+ "normalized": true,
30
+ "rstrip": false,
31
+ "single_word": false
32
+ },
33
+ "use_fast": false
34
+ }
trainer_state.json DELETED
@@ -1,2218 +0,0 @@
1
- {
2
- "best_metric": null,
3
- "best_model_checkpoint": null,
4
- "epoch": 1.8001484046500122,
5
- "global_step": 3639,
6
- "is_hyper_param_search": false,
7
- "is_local_process_zero": true,
8
- "is_world_process_zero": true,
9
- "log_history": [
10
- {
11
- "epoch": 0.0,
12
- "learning_rate": 2.253431117230021e-06,
13
- "loss": 3.9812,
14
- "step": 10
15
- },
16
- {
17
- "epoch": 0.01,
18
- "learning_rate": 3.7916371566987085e-06,
19
- "loss": 0.5473,
20
- "step": 20
21
- },
22
- {
23
- "epoch": 0.01,
24
- "learning_rate": 4.506862234460042e-06,
25
- "loss": 0.3243,
26
- "step": 30
27
- },
28
- {
29
- "epoch": 0.02,
30
- "learning_rate": 4.977968949561282e-06,
31
- "loss": 0.3058,
32
- "step": 40
33
- },
34
- {
35
- "epoch": 0.02,
36
- "learning_rate": 5.3298431961673955e-06,
37
- "loss": 0.2495,
38
- "step": 50
39
- },
40
- {
41
- "epoch": 0.03,
42
- "learning_rate": 5.610809355829126e-06,
43
- "loss": 0.3345,
44
- "step": 60
45
- },
46
- {
47
- "epoch": 0.03,
48
- "learning_rate": 5.844707706136422e-06,
49
- "loss": 0.2784,
50
- "step": 70
51
- },
52
- {
53
- "epoch": 0.04,
54
- "learning_rate": 6.04506827392873e-06,
55
- "loss": 0.2421,
56
- "step": 80
57
- },
58
- {
59
- "epoch": 0.04,
60
- "learning_rate": 6.220313630636092e-06,
61
- "loss": 0.2661,
62
- "step": 90
63
- },
64
- {
65
- "epoch": 0.05,
66
- "learning_rate": 6.376044588167711e-06,
67
- "loss": 0.2831,
68
- "step": 100
69
- },
70
- {
71
- "epoch": 0.05,
72
- "learning_rate": 6.516174989029968e-06,
73
- "loss": 0.2793,
74
- "step": 110
75
- },
76
- {
77
- "epoch": 0.06,
78
- "learning_rate": 6.643547802988659e-06,
79
- "loss": 0.2488,
80
- "step": 120
81
- },
82
- {
83
- "epoch": 0.06,
84
- "learning_rate": 6.7602933516900655e-06,
85
- "loss": 0.2379,
86
- "step": 130
87
- },
88
- {
89
- "epoch": 0.07,
90
- "learning_rate": 6.868049235636083e-06,
91
- "loss": 0.2388,
92
- "step": 140
93
- },
94
- {
95
- "epoch": 0.07,
96
- "learning_rate": 6.968101466168447e-06,
97
- "loss": 0.2273,
98
- "step": 150
99
- },
100
- {
101
- "epoch": 0.08,
102
- "learning_rate": 7.061478425531149e-06,
103
- "loss": 0.2571,
104
- "step": 160
105
- },
106
- {
107
- "epoch": 0.08,
108
- "learning_rate": 7.149015395297812e-06,
109
- "loss": 0.2466,
110
- "step": 170
111
- },
112
- {
113
- "epoch": 0.09,
114
- "learning_rate": 7.231400066791303e-06,
115
- "loss": 0.2315,
116
- "step": 180
117
- },
118
- {
119
- "epoch": 0.09,
120
- "learning_rate": 7.309205386775784e-06,
121
- "loss": 0.2101,
122
- "step": 190
123
- },
124
- {
125
- "epoch": 0.1,
126
- "learning_rate": 7.375715089769526e-06,
127
- "loss": 0.2425,
128
- "step": 200
129
- },
130
- {
131
- "epoch": 0.1,
132
- "learning_rate": 7.44608846772297e-06,
133
- "loss": 0.243,
134
- "step": 210
135
- },
136
- {
137
- "epoch": 0.11,
138
- "learning_rate": 7.513093380702437e-06,
139
- "loss": 0.2521,
140
- "step": 220
141
- },
142
- {
143
- "epoch": 0.11,
144
- "learning_rate": 7.57703761945179e-06,
145
- "loss": 0.2401,
146
- "step": 230
147
- },
148
- {
149
- "epoch": 0.12,
150
- "learning_rate": 7.638188625267883e-06,
151
- "loss": 0.2342,
152
- "step": 240
153
- },
154
- {
155
- "epoch": 0.12,
156
- "learning_rate": 7.696780248446552e-06,
157
- "loss": 0.2439,
158
- "step": 250
159
- },
160
- {
161
- "epoch": 0.13,
162
- "learning_rate": 7.753018148386997e-06,
163
- "loss": 0.2553,
164
- "step": 260
165
- },
166
- {
167
- "epoch": 0.13,
168
- "learning_rate": 7.80708415034011e-06,
169
- "loss": 0.2547,
170
- "step": 270
171
- },
172
- {
173
- "epoch": 0.14,
174
- "learning_rate": 7.859139791732239e-06,
175
- "loss": 0.2316,
176
- "step": 280
177
- },
178
- {
179
- "epoch": 0.14,
180
- "learning_rate": 7.909329232496527e-06,
181
- "loss": 0.2227,
182
- "step": 290
183
- },
184
- {
185
- "epoch": 0.15,
186
- "learning_rate": 7.957781661555314e-06,
187
- "loss": 0.21,
188
- "step": 300
189
- },
190
- {
191
- "epoch": 0.15,
192
- "learning_rate": 8e-06,
193
- "loss": 0.2145,
194
- "step": 310
195
- },
196
- {
197
- "epoch": 0.16,
198
- "learning_rate": 7.98611111111111e-06,
199
- "loss": 0.2101,
200
- "step": 320
201
- },
202
- {
203
- "epoch": 0.16,
204
- "learning_rate": 7.972222222222223e-06,
205
- "loss": 0.1999,
206
- "step": 330
207
- },
208
- {
209
- "epoch": 0.17,
210
- "learning_rate": 7.958333333333333e-06,
211
- "loss": 0.2254,
212
- "step": 340
213
- },
214
- {
215
- "epoch": 0.17,
216
- "learning_rate": 7.944444444444444e-06,
217
- "loss": 0.2597,
218
- "step": 350
219
- },
220
- {
221
- "epoch": 0.18,
222
- "learning_rate": 7.930555555555554e-06,
223
- "loss": 0.1814,
224
- "step": 360
225
- },
226
- {
227
- "epoch": 0.18,
228
- "learning_rate": 7.916666666666667e-06,
229
- "loss": 0.2173,
230
- "step": 370
231
- },
232
- {
233
- "epoch": 0.19,
234
- "learning_rate": 7.902777777777777e-06,
235
- "loss": 0.2136,
236
- "step": 380
237
- },
238
- {
239
- "epoch": 0.19,
240
- "learning_rate": 7.88888888888889e-06,
241
- "loss": 0.2015,
242
- "step": 390
243
- },
244
- {
245
- "epoch": 0.2,
246
- "learning_rate": 7.875e-06,
247
- "loss": 0.2059,
248
- "step": 400
249
- },
250
- {
251
- "epoch": 0.2,
252
- "learning_rate": 7.86111111111111e-06,
253
- "loss": 0.23,
254
- "step": 410
255
- },
256
- {
257
- "epoch": 0.21,
258
- "learning_rate": 7.847222222222221e-06,
259
- "loss": 0.166,
260
- "step": 420
261
- },
262
- {
263
- "epoch": 0.21,
264
- "learning_rate": 7.833333333333333e-06,
265
- "loss": 0.2403,
266
- "step": 430
267
- },
268
- {
269
- "epoch": 0.22,
270
- "learning_rate": 7.819444444444444e-06,
271
- "loss": 0.2111,
272
- "step": 440
273
- },
274
- {
275
- "epoch": 0.22,
276
- "learning_rate": 7.805555555555555e-06,
277
- "loss": 0.2309,
278
- "step": 450
279
- },
280
- {
281
- "epoch": 0.23,
282
- "learning_rate": 7.791666666666667e-06,
283
- "loss": 0.2117,
284
- "step": 460
285
- },
286
- {
287
- "epoch": 0.23,
288
- "learning_rate": 7.777777777777777e-06,
289
- "loss": 0.2085,
290
- "step": 470
291
- },
292
- {
293
- "epoch": 0.24,
294
- "learning_rate": 7.76388888888889e-06,
295
- "loss": 0.1904,
296
- "step": 480
297
- },
298
- {
299
- "epoch": 0.24,
300
- "learning_rate": 7.75e-06,
301
- "loss": 0.1954,
302
- "step": 490
303
- },
304
- {
305
- "epoch": 0.25,
306
- "learning_rate": 7.736111111111111e-06,
307
- "loss": 0.2056,
308
- "step": 500
309
- },
310
- {
311
- "epoch": 0.25,
312
- "learning_rate": 7.722222222222222e-06,
313
- "loss": 0.169,
314
- "step": 510
315
- },
316
- {
317
- "epoch": 0.26,
318
- "learning_rate": 7.708333333333332e-06,
319
- "loss": 0.1814,
320
- "step": 520
321
- },
322
- {
323
- "epoch": 0.26,
324
- "learning_rate": 7.694444444444444e-06,
325
- "loss": 0.21,
326
- "step": 530
327
- },
328
- {
329
- "epoch": 0.27,
330
- "learning_rate": 7.680555555555555e-06,
331
- "loss": 0.2285,
332
- "step": 540
333
- },
334
- {
335
- "epoch": 0.27,
336
- "learning_rate": 7.666666666666667e-06,
337
- "loss": 0.2074,
338
- "step": 550
339
- },
340
- {
341
- "epoch": 0.28,
342
- "learning_rate": 7.652777777777778e-06,
343
- "loss": 0.173,
344
- "step": 560
345
- },
346
- {
347
- "epoch": 0.28,
348
- "learning_rate": 7.638888888888888e-06,
349
- "loss": 0.187,
350
- "step": 570
351
- },
352
- {
353
- "epoch": 0.29,
354
- "learning_rate": 7.625e-06,
355
- "loss": 0.2002,
356
- "step": 580
357
- },
358
- {
359
- "epoch": 0.29,
360
- "learning_rate": 7.6111111111111104e-06,
361
- "loss": 0.2257,
362
- "step": 590
363
- },
364
- {
365
- "epoch": 0.3,
366
- "learning_rate": 7.597222222222222e-06,
367
- "loss": 0.2068,
368
- "step": 600
369
- },
370
- {
371
- "epoch": 0.3,
372
- "learning_rate": 7.5833333333333324e-06,
373
- "loss": 0.1968,
374
- "step": 610
375
- },
376
- {
377
- "epoch": 0.31,
378
- "learning_rate": 7.569444444444445e-06,
379
- "loss": 0.2122,
380
- "step": 620
381
- },
382
- {
383
- "epoch": 0.31,
384
- "learning_rate": 7.555555555555555e-06,
385
- "loss": 0.2306,
386
- "step": 630
387
- },
388
- {
389
- "epoch": 0.32,
390
- "learning_rate": 7.541666666666667e-06,
391
- "loss": 0.2108,
392
- "step": 640
393
- },
394
- {
395
- "epoch": 0.32,
396
- "learning_rate": 7.527777777777777e-06,
397
- "loss": 0.1923,
398
- "step": 650
399
- },
400
- {
401
- "epoch": 0.33,
402
- "learning_rate": 7.513888888888889e-06,
403
- "loss": 0.1989,
404
- "step": 660
405
- },
406
- {
407
- "epoch": 0.33,
408
- "learning_rate": 7.499999999999999e-06,
409
- "loss": 0.2179,
410
- "step": 670
411
- },
412
- {
413
- "epoch": 0.34,
414
- "learning_rate": 7.486111111111111e-06,
415
- "loss": 0.1915,
416
- "step": 680
417
- },
418
- {
419
- "epoch": 0.34,
420
- "learning_rate": 7.472222222222222e-06,
421
- "loss": 0.1965,
422
- "step": 690
423
- },
424
- {
425
- "epoch": 0.35,
426
- "learning_rate": 7.458333333333333e-06,
427
- "loss": 0.2425,
428
- "step": 700
429
- },
430
- {
431
- "epoch": 0.35,
432
- "learning_rate": 7.444444444444444e-06,
433
- "loss": 0.1933,
434
- "step": 710
435
- },
436
- {
437
- "epoch": 0.36,
438
- "learning_rate": 7.430555555555555e-06,
439
- "loss": 0.2314,
440
- "step": 720
441
- },
442
- {
443
- "epoch": 0.36,
444
- "learning_rate": 7.416666666666666e-06,
445
- "loss": 0.1901,
446
- "step": 730
447
- },
448
- {
449
- "epoch": 0.37,
450
- "learning_rate": 7.402777777777778e-06,
451
- "loss": 0.1852,
452
- "step": 740
453
- },
454
- {
455
- "epoch": 0.37,
456
- "learning_rate": 7.388888888888889e-06,
457
- "loss": 0.201,
458
- "step": 750
459
- },
460
- {
461
- "epoch": 0.38,
462
- "learning_rate": 7.375e-06,
463
- "loss": 0.1636,
464
- "step": 760
465
- },
466
- {
467
- "epoch": 0.38,
468
- "learning_rate": 7.36111111111111e-06,
469
- "loss": 0.184,
470
- "step": 770
471
- },
472
- {
473
- "epoch": 0.39,
474
- "learning_rate": 7.347222222222222e-06,
475
- "loss": 0.1618,
476
- "step": 780
477
- },
478
- {
479
- "epoch": 0.39,
480
- "learning_rate": 7.333333333333332e-06,
481
- "loss": 0.1867,
482
- "step": 790
483
- },
484
- {
485
- "epoch": 0.4,
486
- "learning_rate": 7.3194444444444446e-06,
487
- "loss": 0.1877,
488
- "step": 800
489
- },
490
- {
491
- "epoch": 0.4,
492
- "learning_rate": 7.305555555555555e-06,
493
- "loss": 0.1569,
494
- "step": 810
495
- },
496
- {
497
- "epoch": 0.41,
498
- "learning_rate": 7.291666666666667e-06,
499
- "loss": 0.1651,
500
- "step": 820
501
- },
502
- {
503
- "epoch": 0.41,
504
- "learning_rate": 7.277777777777777e-06,
505
- "loss": 0.215,
506
- "step": 830
507
- },
508
- {
509
- "epoch": 0.42,
510
- "learning_rate": 7.263888888888889e-06,
511
- "loss": 0.1609,
512
- "step": 840
513
- },
514
- {
515
- "epoch": 0.42,
516
- "learning_rate": 7.25e-06,
517
- "loss": 0.1935,
518
- "step": 850
519
- },
520
- {
521
- "epoch": 0.43,
522
- "learning_rate": 7.236111111111111e-06,
523
- "loss": 0.1688,
524
- "step": 860
525
- },
526
- {
527
- "epoch": 0.43,
528
- "learning_rate": 7.222222222222222e-06,
529
- "loss": 0.1645,
530
- "step": 870
531
- },
532
- {
533
- "epoch": 0.44,
534
- "learning_rate": 7.208333333333333e-06,
535
- "loss": 0.162,
536
- "step": 880
537
- },
538
- {
539
- "epoch": 0.44,
540
- "learning_rate": 7.194444444444444e-06,
541
- "loss": 0.1467,
542
- "step": 890
543
- },
544
- {
545
- "epoch": 0.45,
546
- "learning_rate": 7.180555555555555e-06,
547
- "loss": 0.1579,
548
- "step": 900
549
- },
550
- {
551
- "epoch": 0.45,
552
- "learning_rate": 7.166666666666667e-06,
553
- "loss": 0.1774,
554
- "step": 910
555
- },
556
- {
557
- "epoch": 0.46,
558
- "learning_rate": 7.1527777777777775e-06,
559
- "loss": 0.1931,
560
- "step": 920
561
- },
562
- {
563
- "epoch": 0.46,
564
- "learning_rate": 7.138888888888889e-06,
565
- "loss": 0.1505,
566
- "step": 930
567
- },
568
- {
569
- "epoch": 0.47,
570
- "learning_rate": 7.1249999999999995e-06,
571
- "loss": 0.1632,
572
- "step": 940
573
- },
574
- {
575
- "epoch": 0.47,
576
- "learning_rate": 7.11111111111111e-06,
577
- "loss": 0.169,
578
- "step": 950
579
- },
580
- {
581
- "epoch": 0.47,
582
- "learning_rate": 7.097222222222222e-06,
583
- "loss": 0.1762,
584
- "step": 960
585
- },
586
- {
587
- "epoch": 0.48,
588
- "learning_rate": 7.083333333333333e-06,
589
- "loss": 0.1838,
590
- "step": 970
591
- },
592
- {
593
- "epoch": 0.48,
594
- "learning_rate": 7.0694444444444444e-06,
595
- "loss": 0.1622,
596
- "step": 980
597
- },
598
- {
599
- "epoch": 0.49,
600
- "learning_rate": 7.055555555555555e-06,
601
- "loss": 0.1562,
602
- "step": 990
603
- },
604
- {
605
- "epoch": 0.49,
606
- "learning_rate": 7.0416666666666664e-06,
607
- "loss": 0.1475,
608
- "step": 1000
609
- },
610
- {
611
- "epoch": 0.5,
612
- "learning_rate": 7.027777777777777e-06,
613
- "loss": 0.1557,
614
- "step": 1010
615
- },
616
- {
617
- "epoch": 0.5,
618
- "learning_rate": 7.013888888888889e-06,
619
- "loss": 0.1851,
620
- "step": 1020
621
- },
622
- {
623
- "epoch": 0.51,
624
- "learning_rate": 7e-06,
625
- "loss": 0.1607,
626
- "step": 1030
627
- },
628
- {
629
- "epoch": 0.51,
630
- "learning_rate": 6.9861111111111105e-06,
631
- "loss": 0.1541,
632
- "step": 1040
633
- },
634
- {
635
- "epoch": 0.52,
636
- "learning_rate": 6.972222222222222e-06,
637
- "loss": 0.1509,
638
- "step": 1050
639
- },
640
- {
641
- "epoch": 0.52,
642
- "learning_rate": 6.9583333333333325e-06,
643
- "loss": 0.1754,
644
- "step": 1060
645
- },
646
- {
647
- "epoch": 0.53,
648
- "learning_rate": 6.944444444444444e-06,
649
- "loss": 0.205,
650
- "step": 1070
651
- },
652
- {
653
- "epoch": 0.53,
654
- "learning_rate": 6.930555555555555e-06,
655
- "loss": 0.2037,
656
- "step": 1080
657
- },
658
- {
659
- "epoch": 0.54,
660
- "learning_rate": 6.916666666666667e-06,
661
- "loss": 0.1424,
662
- "step": 1090
663
- },
664
- {
665
- "epoch": 0.54,
666
- "learning_rate": 6.902777777777777e-06,
667
- "loss": 0.1594,
668
- "step": 1100
669
- },
670
- {
671
- "epoch": 0.55,
672
- "learning_rate": 6.888888888888889e-06,
673
- "loss": 0.1838,
674
- "step": 1110
675
- },
676
- {
677
- "epoch": 0.55,
678
- "learning_rate": 6.874999999999999e-06,
679
- "loss": 0.1626,
680
- "step": 1120
681
- },
682
- {
683
- "epoch": 0.56,
684
- "learning_rate": 6.86111111111111e-06,
685
- "loss": 0.1689,
686
- "step": 1130
687
- },
688
- {
689
- "epoch": 0.56,
690
- "learning_rate": 6.847222222222222e-06,
691
- "loss": 0.1548,
692
- "step": 1140
693
- },
694
- {
695
- "epoch": 0.57,
696
- "learning_rate": 6.833333333333333e-06,
697
- "loss": 0.1435,
698
- "step": 1150
699
- },
700
- {
701
- "epoch": 0.57,
702
- "learning_rate": 6.819444444444444e-06,
703
- "loss": 0.1939,
704
- "step": 1160
705
- },
706
- {
707
- "epoch": 0.58,
708
- "learning_rate": 6.805555555555555e-06,
709
- "loss": 0.1802,
710
- "step": 1170
711
- },
712
- {
713
- "epoch": 0.58,
714
- "learning_rate": 6.791666666666666e-06,
715
- "loss": 0.1865,
716
- "step": 1180
717
- },
718
- {
719
- "epoch": 0.59,
720
- "learning_rate": 6.777777777777778e-06,
721
- "loss": 0.18,
722
- "step": 1190
723
- },
724
- {
725
- "epoch": 0.59,
726
- "learning_rate": 6.763888888888889e-06,
727
- "loss": 0.1862,
728
- "step": 1200
729
- },
730
- {
731
- "epoch": 0.6,
732
- "learning_rate": 6.75e-06,
733
- "loss": 0.183,
734
- "step": 1210
735
- },
736
- {
737
- "epoch": 0.6,
738
- "eval_loss": 0.234619140625,
739
- "eval_runtime": 601.4014,
740
- "eval_samples_per_second": 39.832,
741
- "eval_steps_per_second": 4.98,
742
- "step": 1213
743
- },
744
- {
745
- "epoch": 0.6,
746
- "learning_rate": 6.73611111111111e-06,
747
- "loss": 0.1954,
748
- "step": 1220
749
- },
750
- {
751
- "epoch": 0.61,
752
- "learning_rate": 6.722222222222222e-06,
753
- "loss": 0.1433,
754
- "step": 1230
755
- },
756
- {
757
- "epoch": 0.61,
758
- "learning_rate": 6.708333333333332e-06,
759
- "loss": 0.1732,
760
- "step": 1240
761
- },
762
- {
763
- "epoch": 0.62,
764
- "learning_rate": 6.694444444444445e-06,
765
- "loss": 0.1985,
766
- "step": 1250
767
- },
768
- {
769
- "epoch": 0.62,
770
- "learning_rate": 6.680555555555555e-06,
771
- "loss": 0.1786,
772
- "step": 1260
773
- },
774
- {
775
- "epoch": 0.63,
776
- "learning_rate": 6.666666666666667e-06,
777
- "loss": 0.167,
778
- "step": 1270
779
- },
780
- {
781
- "epoch": 0.63,
782
- "learning_rate": 6.652777777777777e-06,
783
- "loss": 0.1548,
784
- "step": 1280
785
- },
786
- {
787
- "epoch": 0.64,
788
- "learning_rate": 6.638888888888889e-06,
789
- "loss": 0.1559,
790
- "step": 1290
791
- },
792
- {
793
- "epoch": 0.64,
794
- "learning_rate": 6.625e-06,
795
- "loss": 0.176,
796
- "step": 1300
797
- },
798
- {
799
- "epoch": 0.65,
800
- "learning_rate": 6.611111111111111e-06,
801
- "loss": 0.1687,
802
- "step": 1310
803
- },
804
- {
805
- "epoch": 0.65,
806
- "learning_rate": 6.597222222222222e-06,
807
- "loss": 0.1497,
808
- "step": 1320
809
- },
810
- {
811
- "epoch": 0.66,
812
- "learning_rate": 6.583333333333333e-06,
813
- "loss": 0.1891,
814
- "step": 1330
815
- },
816
- {
817
- "epoch": 0.66,
818
- "learning_rate": 6.569444444444444e-06,
819
- "loss": 0.1783,
820
- "step": 1340
821
- },
822
- {
823
- "epoch": 0.67,
824
- "learning_rate": 6.555555555555555e-06,
825
- "loss": 0.168,
826
- "step": 1350
827
- },
828
- {
829
- "epoch": 0.67,
830
- "learning_rate": 6.541666666666667e-06,
831
- "loss": 0.1871,
832
- "step": 1360
833
- },
834
- {
835
- "epoch": 0.68,
836
- "learning_rate": 6.527777777777778e-06,
837
- "loss": 0.1601,
838
- "step": 1370
839
- },
840
- {
841
- "epoch": 0.68,
842
- "learning_rate": 6.513888888888889e-06,
843
- "loss": 0.1466,
844
- "step": 1380
845
- },
846
- {
847
- "epoch": 0.69,
848
- "learning_rate": 6.5e-06,
849
- "loss": 0.1799,
850
- "step": 1390
851
- },
852
- {
853
- "epoch": 0.69,
854
- "learning_rate": 6.48611111111111e-06,
855
- "loss": 0.1448,
856
- "step": 1400
857
- },
858
- {
859
- "epoch": 0.7,
860
- "learning_rate": 6.472222222222222e-06,
861
- "loss": 0.1459,
862
- "step": 1410
863
- },
864
- {
865
- "epoch": 0.7,
866
- "learning_rate": 6.458333333333333e-06,
867
- "loss": 0.1677,
868
- "step": 1420
869
- },
870
- {
871
- "epoch": 0.71,
872
- "learning_rate": 6.4444444444444445e-06,
873
- "loss": 0.1878,
874
- "step": 1430
875
- },
876
- {
877
- "epoch": 0.71,
878
- "learning_rate": 6.430555555555555e-06,
879
- "loss": 0.1494,
880
- "step": 1440
881
- },
882
- {
883
- "epoch": 0.72,
884
- "learning_rate": 6.4166666666666665e-06,
885
- "loss": 0.1406,
886
- "step": 1450
887
- },
888
- {
889
- "epoch": 0.72,
890
- "learning_rate": 6.402777777777777e-06,
891
- "loss": 0.1538,
892
- "step": 1460
893
- },
894
- {
895
- "epoch": 0.73,
896
- "learning_rate": 6.390277777777778e-06,
897
- "loss": 0.169,
898
- "step": 1470
899
- },
900
- {
901
- "epoch": 0.73,
902
- "learning_rate": 6.376388888888889e-06,
903
- "loss": 0.1444,
904
- "step": 1480
905
- },
906
- {
907
- "epoch": 0.74,
908
- "learning_rate": 6.3625e-06,
909
- "loss": 0.1561,
910
- "step": 1490
911
- },
912
- {
913
- "epoch": 0.74,
914
- "learning_rate": 6.348611111111111e-06,
915
- "loss": 0.1493,
916
- "step": 1500
917
- },
918
- {
919
- "epoch": 0.75,
920
- "learning_rate": 6.334722222222222e-06,
921
- "loss": 0.1547,
922
- "step": 1510
923
- },
924
- {
925
- "epoch": 0.75,
926
- "learning_rate": 6.320833333333333e-06,
927
- "loss": 0.1749,
928
- "step": 1520
929
- },
930
- {
931
- "epoch": 0.76,
932
- "learning_rate": 6.3069444444444445e-06,
933
- "loss": 0.1903,
934
- "step": 1530
935
- },
936
- {
937
- "epoch": 0.76,
938
- "learning_rate": 6.293055555555555e-06,
939
- "loss": 0.18,
940
- "step": 1540
941
- },
942
- {
943
- "epoch": 0.77,
944
- "learning_rate": 6.2791666666666665e-06,
945
- "loss": 0.1491,
946
- "step": 1550
947
- },
948
- {
949
- "epoch": 0.77,
950
- "learning_rate": 6.265277777777777e-06,
951
- "loss": 0.1777,
952
- "step": 1560
953
- },
954
- {
955
- "epoch": 0.78,
956
- "learning_rate": 6.2513888888888886e-06,
957
- "loss": 0.1226,
958
- "step": 1570
959
- },
960
- {
961
- "epoch": 0.78,
962
- "learning_rate": 6.237499999999999e-06,
963
- "loss": 0.1822,
964
- "step": 1580
965
- },
966
- {
967
- "epoch": 0.79,
968
- "learning_rate": 6.2236111111111114e-06,
969
- "loss": 0.143,
970
- "step": 1590
971
- },
972
- {
973
- "epoch": 0.79,
974
- "learning_rate": 6.209722222222222e-06,
975
- "loss": 0.1411,
976
- "step": 1600
977
- },
978
- {
979
- "epoch": 0.8,
980
- "learning_rate": 6.1958333333333334e-06,
981
- "loss": 0.1452,
982
- "step": 1610
983
- },
984
- {
985
- "epoch": 0.8,
986
- "learning_rate": 6.181944444444444e-06,
987
- "loss": 0.1673,
988
- "step": 1620
989
- },
990
- {
991
- "epoch": 0.81,
992
- "learning_rate": 6.169444444444444e-06,
993
- "loss": 0.1706,
994
- "step": 1630
995
- },
996
- {
997
- "epoch": 0.81,
998
- "learning_rate": 6.155555555555556e-06,
999
- "loss": 0.1755,
1000
- "step": 1640
1001
- },
1002
- {
1003
- "epoch": 0.82,
1004
- "learning_rate": 6.141666666666667e-06,
1005
- "loss": 0.1831,
1006
- "step": 1650
1007
- },
1008
- {
1009
- "epoch": 0.82,
1010
- "learning_rate": 6.127777777777778e-06,
1011
- "loss": 0.1402,
1012
- "step": 1660
1013
- },
1014
- {
1015
- "epoch": 0.83,
1016
- "learning_rate": 6.113888888888889e-06,
1017
- "loss": 0.1434,
1018
- "step": 1670
1019
- },
1020
- {
1021
- "epoch": 0.83,
1022
- "learning_rate": 6.099999999999999e-06,
1023
- "loss": 0.1347,
1024
- "step": 1680
1025
- },
1026
- {
1027
- "epoch": 0.84,
1028
- "learning_rate": 6.086111111111111e-06,
1029
- "loss": 0.1614,
1030
- "step": 1690
1031
- },
1032
- {
1033
- "epoch": 0.84,
1034
- "learning_rate": 6.072222222222222e-06,
1035
- "loss": 0.1533,
1036
- "step": 1700
1037
- },
1038
- {
1039
- "epoch": 0.85,
1040
- "learning_rate": 6.0583333333333335e-06,
1041
- "loss": 0.1666,
1042
- "step": 1710
1043
- },
1044
- {
1045
- "epoch": 0.85,
1046
- "learning_rate": 6.044444444444444e-06,
1047
- "loss": 0.1725,
1048
- "step": 1720
1049
- },
1050
- {
1051
- "epoch": 0.86,
1052
- "learning_rate": 6.0305555555555555e-06,
1053
- "loss": 0.1819,
1054
- "step": 1730
1055
- },
1056
- {
1057
- "epoch": 0.86,
1058
- "learning_rate": 6.016666666666666e-06,
1059
- "loss": 0.1739,
1060
- "step": 1740
1061
- },
1062
- {
1063
- "epoch": 0.87,
1064
- "learning_rate": 6.0027777777777775e-06,
1065
- "loss": 0.1573,
1066
- "step": 1750
1067
- },
1068
- {
1069
- "epoch": 0.87,
1070
- "learning_rate": 5.988888888888889e-06,
1071
- "loss": 0.1458,
1072
- "step": 1760
1073
- },
1074
- {
1075
- "epoch": 0.88,
1076
- "learning_rate": 5.9749999999999995e-06,
1077
- "loss": 0.1243,
1078
- "step": 1770
1079
- },
1080
- {
1081
- "epoch": 0.88,
1082
- "learning_rate": 5.961111111111111e-06,
1083
- "loss": 0.1657,
1084
- "step": 1780
1085
- },
1086
- {
1087
- "epoch": 0.89,
1088
- "learning_rate": 5.9472222222222216e-06,
1089
- "loss": 0.1871,
1090
- "step": 1790
1091
- },
1092
- {
1093
- "epoch": 0.89,
1094
- "learning_rate": 5.933333333333333e-06,
1095
- "loss": 0.1365,
1096
- "step": 1800
1097
- },
1098
- {
1099
- "epoch": 0.9,
1100
- "learning_rate": 5.9194444444444444e-06,
1101
- "loss": 0.0987,
1102
- "step": 1810
1103
- },
1104
- {
1105
- "epoch": 0.9,
1106
- "learning_rate": 5.905555555555556e-06,
1107
- "loss": 0.1396,
1108
- "step": 1820
1109
- },
1110
- {
1111
- "epoch": 0.91,
1112
- "learning_rate": 5.8916666666666664e-06,
1113
- "loss": 0.1693,
1114
- "step": 1830
1115
- },
1116
- {
1117
- "epoch": 0.91,
1118
- "learning_rate": 5.877777777777778e-06,
1119
- "loss": 0.1391,
1120
- "step": 1840
1121
- },
1122
- {
1123
- "epoch": 0.92,
1124
- "learning_rate": 5.8638888888888885e-06,
1125
- "loss": 0.1439,
1126
- "step": 1850
1127
- },
1128
- {
1129
- "epoch": 0.92,
1130
- "learning_rate": 5.849999999999999e-06,
1131
- "loss": 0.1503,
1132
- "step": 1860
1133
- },
1134
- {
1135
- "epoch": 0.93,
1136
- "learning_rate": 5.836111111111111e-06,
1137
- "loss": 0.1359,
1138
- "step": 1870
1139
- },
1140
- {
1141
- "epoch": 0.93,
1142
- "learning_rate": 5.822222222222222e-06,
1143
- "loss": 0.1423,
1144
- "step": 1880
1145
- },
1146
- {
1147
- "epoch": 0.93,
1148
- "learning_rate": 5.808333333333333e-06,
1149
- "loss": 0.1317,
1150
- "step": 1890
1151
- },
1152
- {
1153
- "epoch": 0.94,
1154
- "learning_rate": 5.794444444444444e-06,
1155
- "loss": 0.1431,
1156
- "step": 1900
1157
- },
1158
- {
1159
- "epoch": 0.94,
1160
- "learning_rate": 5.780555555555555e-06,
1161
- "loss": 0.1235,
1162
- "step": 1910
1163
- },
1164
- {
1165
- "epoch": 0.95,
1166
- "learning_rate": 5.766666666666666e-06,
1167
- "loss": 0.1397,
1168
- "step": 1920
1169
- },
1170
- {
1171
- "epoch": 0.95,
1172
- "learning_rate": 5.752777777777778e-06,
1173
- "loss": 0.1375,
1174
- "step": 1930
1175
- },
1176
- {
1177
- "epoch": 0.96,
1178
- "learning_rate": 5.738888888888889e-06,
1179
- "loss": 0.1341,
1180
- "step": 1940
1181
- },
1182
- {
1183
- "epoch": 0.96,
1184
- "learning_rate": 5.724999999999999e-06,
1185
- "loss": 0.1408,
1186
- "step": 1950
1187
- },
1188
- {
1189
- "epoch": 0.97,
1190
- "learning_rate": 5.711111111111111e-06,
1191
- "loss": 0.1766,
1192
- "step": 1960
1193
- },
1194
- {
1195
- "epoch": 0.97,
1196
- "learning_rate": 5.697222222222221e-06,
1197
- "loss": 0.177,
1198
- "step": 1970
1199
- },
1200
- {
1201
- "epoch": 0.98,
1202
- "learning_rate": 5.683333333333334e-06,
1203
- "loss": 0.1586,
1204
- "step": 1980
1205
- },
1206
- {
1207
- "epoch": 0.98,
1208
- "learning_rate": 5.669444444444444e-06,
1209
- "loss": 0.1447,
1210
- "step": 1990
1211
- },
1212
- {
1213
- "epoch": 0.99,
1214
- "learning_rate": 5.655555555555556e-06,
1215
- "loss": 0.1452,
1216
- "step": 2000
1217
- },
1218
- {
1219
- "epoch": 0.99,
1220
- "learning_rate": 5.641666666666666e-06,
1221
- "loss": 0.1078,
1222
- "step": 2010
1223
- },
1224
- {
1225
- "epoch": 1.0,
1226
- "learning_rate": 5.627777777777778e-06,
1227
- "loss": 0.1133,
1228
- "step": 2020
1229
- },
1230
- {
1231
- "epoch": 1.0,
1232
- "learning_rate": 5.613888888888888e-06,
1233
- "loss": 0.1288,
1234
- "step": 2030
1235
- },
1236
- {
1237
- "epoch": 1.01,
1238
- "learning_rate": 5.6e-06,
1239
- "loss": 0.1128,
1240
- "step": 2040
1241
- },
1242
- {
1243
- "epoch": 1.01,
1244
- "learning_rate": 5.586111111111111e-06,
1245
- "loss": 0.1298,
1246
- "step": 2050
1247
- },
1248
- {
1249
- "epoch": 1.02,
1250
- "learning_rate": 5.572222222222222e-06,
1251
- "loss": 0.1189,
1252
- "step": 2060
1253
- },
1254
- {
1255
- "epoch": 1.02,
1256
- "learning_rate": 5.558333333333333e-06,
1257
- "loss": 0.1478,
1258
- "step": 2070
1259
- },
1260
- {
1261
- "epoch": 1.03,
1262
- "learning_rate": 5.544444444444444e-06,
1263
- "loss": 0.1179,
1264
- "step": 2080
1265
- },
1266
- {
1267
- "epoch": 1.03,
1268
- "learning_rate": 5.530555555555556e-06,
1269
- "loss": 0.1119,
1270
- "step": 2090
1271
- },
1272
- {
1273
- "epoch": 1.04,
1274
- "learning_rate": 5.516666666666667e-06,
1275
- "loss": 0.1355,
1276
- "step": 2100
1277
- },
1278
- {
1279
- "epoch": 1.04,
1280
- "learning_rate": 5.502777777777778e-06,
1281
- "loss": 0.1105,
1282
- "step": 2110
1283
- },
1284
- {
1285
- "epoch": 1.05,
1286
- "learning_rate": 5.488888888888889e-06,
1287
- "loss": 0.1289,
1288
- "step": 2120
1289
- },
1290
- {
1291
- "epoch": 1.05,
1292
- "learning_rate": 5.474999999999999e-06,
1293
- "loss": 0.1127,
1294
- "step": 2130
1295
- },
1296
- {
1297
- "epoch": 1.06,
1298
- "learning_rate": 5.461111111111111e-06,
1299
- "loss": 0.1814,
1300
- "step": 2140
1301
- },
1302
- {
1303
- "epoch": 1.06,
1304
- "learning_rate": 5.447222222222222e-06,
1305
- "loss": 0.1554,
1306
- "step": 2150
1307
- },
1308
- {
1309
- "epoch": 1.07,
1310
- "learning_rate": 5.4333333333333335e-06,
1311
- "loss": 0.1038,
1312
- "step": 2160
1313
- },
1314
- {
1315
- "epoch": 1.07,
1316
- "learning_rate": 5.419444444444444e-06,
1317
- "loss": 0.1085,
1318
- "step": 2170
1319
- },
1320
- {
1321
- "epoch": 1.08,
1322
- "learning_rate": 5.4055555555555556e-06,
1323
- "loss": 0.1364,
1324
- "step": 2180
1325
- },
1326
- {
1327
- "epoch": 1.08,
1328
- "learning_rate": 5.391666666666666e-06,
1329
- "loss": 0.1089,
1330
- "step": 2190
1331
- },
1332
- {
1333
- "epoch": 1.09,
1334
- "learning_rate": 5.377777777777778e-06,
1335
- "loss": 0.1484,
1336
- "step": 2200
1337
- },
1338
- {
1339
- "epoch": 1.09,
1340
- "learning_rate": 5.363888888888889e-06,
1341
- "loss": 0.1254,
1342
- "step": 2210
1343
- },
1344
- {
1345
- "epoch": 1.1,
1346
- "learning_rate": 5.35e-06,
1347
- "loss": 0.1228,
1348
- "step": 2220
1349
- },
1350
- {
1351
- "epoch": 1.1,
1352
- "learning_rate": 5.336111111111111e-06,
1353
- "loss": 0.154,
1354
- "step": 2230
1355
- },
1356
- {
1357
- "epoch": 1.11,
1358
- "learning_rate": 5.322222222222222e-06,
1359
- "loss": 0.1745,
1360
- "step": 2240
1361
- },
1362
- {
1363
- "epoch": 1.11,
1364
- "learning_rate": 5.308333333333333e-06,
1365
- "loss": 0.1031,
1366
- "step": 2250
1367
- },
1368
- {
1369
- "epoch": 1.12,
1370
- "learning_rate": 5.294444444444444e-06,
1371
- "loss": 0.1359,
1372
- "step": 2260
1373
- },
1374
- {
1375
- "epoch": 1.12,
1376
- "learning_rate": 5.280555555555556e-06,
1377
- "loss": 0.1194,
1378
- "step": 2270
1379
- },
1380
- {
1381
- "epoch": 1.13,
1382
- "learning_rate": 5.2666666666666665e-06,
1383
- "loss": 0.1398,
1384
- "step": 2280
1385
- },
1386
- {
1387
- "epoch": 1.13,
1388
- "learning_rate": 5.252777777777778e-06,
1389
- "loss": 0.1277,
1390
- "step": 2290
1391
- },
1392
- {
1393
- "epoch": 1.14,
1394
- "learning_rate": 5.2388888888888885e-06,
1395
- "loss": 0.1121,
1396
- "step": 2300
1397
- },
1398
- {
1399
- "epoch": 1.14,
1400
- "learning_rate": 5.224999999999999e-06,
1401
- "loss": 0.1067,
1402
- "step": 2310
1403
- },
1404
- {
1405
- "epoch": 1.15,
1406
- "learning_rate": 5.211111111111111e-06,
1407
- "loss": 0.1101,
1408
- "step": 2320
1409
- },
1410
- {
1411
- "epoch": 1.15,
1412
- "learning_rate": 5.197222222222222e-06,
1413
- "loss": 0.1192,
1414
- "step": 2330
1415
- },
1416
- {
1417
- "epoch": 1.16,
1418
- "learning_rate": 5.183333333333333e-06,
1419
- "loss": 0.1055,
1420
- "step": 2340
1421
- },
1422
- {
1423
- "epoch": 1.16,
1424
- "learning_rate": 5.169444444444444e-06,
1425
- "loss": 0.1232,
1426
- "step": 2350
1427
- },
1428
- {
1429
- "epoch": 1.17,
1430
- "learning_rate": 5.155555555555555e-06,
1431
- "loss": 0.1096,
1432
- "step": 2360
1433
- },
1434
- {
1435
- "epoch": 1.17,
1436
- "learning_rate": 5.141666666666666e-06,
1437
- "loss": 0.0831,
1438
- "step": 2370
1439
- },
1440
- {
1441
- "epoch": 1.18,
1442
- "learning_rate": 5.127777777777778e-06,
1443
- "loss": 0.149,
1444
- "step": 2380
1445
- },
1446
- {
1447
- "epoch": 1.18,
1448
- "learning_rate": 5.113888888888889e-06,
1449
- "loss": 0.1328,
1450
- "step": 2390
1451
- },
1452
- {
1453
- "epoch": 1.19,
1454
- "learning_rate": 5.0999999999999995e-06,
1455
- "loss": 0.1193,
1456
- "step": 2400
1457
- },
1458
- {
1459
- "epoch": 1.19,
1460
- "learning_rate": 5.086111111111111e-06,
1461
- "loss": 0.1222,
1462
- "step": 2410
1463
- },
1464
- {
1465
- "epoch": 1.2,
1466
- "learning_rate": 5.0722222222222215e-06,
1467
- "loss": 0.1395,
1468
- "step": 2420
1469
- },
1470
- {
1471
- "epoch": 1.2,
1472
- "eval_loss": 0.26611328125,
1473
- "eval_runtime": 595.4698,
1474
- "eval_samples_per_second": 40.229,
1475
- "eval_steps_per_second": 5.03,
1476
- "step": 2426
1477
- },
1478
- {
1479
- "epoch": 1.2,
1480
- "learning_rate": 5.058333333333334e-06,
1481
- "loss": 0.1261,
1482
- "step": 2430
1483
- },
1484
- {
1485
- "epoch": 1.21,
1486
- "learning_rate": 5.044444444444444e-06,
1487
- "loss": 0.1416,
1488
- "step": 2440
1489
- },
1490
- {
1491
- "epoch": 1.21,
1492
- "learning_rate": 5.030555555555556e-06,
1493
- "loss": 0.1453,
1494
- "step": 2450
1495
- },
1496
- {
1497
- "epoch": 1.22,
1498
- "learning_rate": 5.016666666666666e-06,
1499
- "loss": 0.114,
1500
- "step": 2460
1501
- },
1502
- {
1503
- "epoch": 1.22,
1504
- "learning_rate": 5.002777777777778e-06,
1505
- "loss": 0.133,
1506
- "step": 2470
1507
- },
1508
- {
1509
- "epoch": 1.23,
1510
- "learning_rate": 4.988888888888888e-06,
1511
- "loss": 0.1207,
1512
- "step": 2480
1513
- },
1514
- {
1515
- "epoch": 1.23,
1516
- "learning_rate": 4.975e-06,
1517
- "loss": 0.1123,
1518
- "step": 2490
1519
- },
1520
- {
1521
- "epoch": 1.24,
1522
- "learning_rate": 4.961111111111111e-06,
1523
- "loss": 0.1039,
1524
- "step": 2500
1525
- },
1526
- {
1527
- "epoch": 1.24,
1528
- "learning_rate": 4.947222222222222e-06,
1529
- "loss": 0.091,
1530
- "step": 2510
1531
- },
1532
- {
1533
- "epoch": 1.25,
1534
- "learning_rate": 4.933333333333333e-06,
1535
- "loss": 0.1588,
1536
- "step": 2520
1537
- },
1538
- {
1539
- "epoch": 1.25,
1540
- "learning_rate": 4.919444444444444e-06,
1541
- "loss": 0.098,
1542
- "step": 2530
1543
- },
1544
- {
1545
- "epoch": 1.26,
1546
- "learning_rate": 4.905555555555555e-06,
1547
- "loss": 0.1033,
1548
- "step": 2540
1549
- },
1550
- {
1551
- "epoch": 1.26,
1552
- "learning_rate": 4.891666666666667e-06,
1553
- "loss": 0.1473,
1554
- "step": 2550
1555
- },
1556
- {
1557
- "epoch": 1.27,
1558
- "learning_rate": 4.877777777777778e-06,
1559
- "loss": 0.1331,
1560
- "step": 2560
1561
- },
1562
- {
1563
- "epoch": 1.27,
1564
- "learning_rate": 4.863888888888889e-06,
1565
- "loss": 0.1196,
1566
- "step": 2570
1567
- },
1568
- {
1569
- "epoch": 1.28,
1570
- "learning_rate": 4.849999999999999e-06,
1571
- "loss": 0.1379,
1572
- "step": 2580
1573
- },
1574
- {
1575
- "epoch": 1.28,
1576
- "learning_rate": 4.836111111111111e-06,
1577
- "loss": 0.124,
1578
- "step": 2590
1579
- },
1580
- {
1581
- "epoch": 1.29,
1582
- "learning_rate": 4.822222222222222e-06,
1583
- "loss": 0.1058,
1584
- "step": 2600
1585
- },
1586
- {
1587
- "epoch": 1.29,
1588
- "learning_rate": 4.808333333333334e-06,
1589
- "loss": 0.1132,
1590
- "step": 2610
1591
- },
1592
- {
1593
- "epoch": 1.3,
1594
- "learning_rate": 4.794444444444444e-06,
1595
- "loss": 0.1077,
1596
- "step": 2620
1597
- },
1598
- {
1599
- "epoch": 1.3,
1600
- "learning_rate": 4.780555555555556e-06,
1601
- "loss": 0.1266,
1602
- "step": 2630
1603
- },
1604
- {
1605
- "epoch": 1.31,
1606
- "learning_rate": 4.766666666666666e-06,
1607
- "loss": 0.1292,
1608
- "step": 2640
1609
- },
1610
- {
1611
- "epoch": 1.31,
1612
- "learning_rate": 4.752777777777778e-06,
1613
- "loss": 0.0969,
1614
- "step": 2650
1615
- },
1616
- {
1617
- "epoch": 1.32,
1618
- "learning_rate": 4.738888888888889e-06,
1619
- "loss": 0.0946,
1620
- "step": 2660
1621
- },
1622
- {
1623
- "epoch": 1.32,
1624
- "learning_rate": 4.725e-06,
1625
- "loss": 0.1479,
1626
- "step": 2670
1627
- },
1628
- {
1629
- "epoch": 1.33,
1630
- "learning_rate": 4.711111111111111e-06,
1631
- "loss": 0.133,
1632
- "step": 2680
1633
- },
1634
- {
1635
- "epoch": 1.33,
1636
- "learning_rate": 4.697222222222222e-06,
1637
- "loss": 0.1118,
1638
- "step": 2690
1639
- },
1640
- {
1641
- "epoch": 1.34,
1642
- "learning_rate": 4.683333333333333e-06,
1643
- "loss": 0.1362,
1644
- "step": 2700
1645
- },
1646
- {
1647
- "epoch": 1.34,
1648
- "learning_rate": 4.669444444444444e-06,
1649
- "loss": 0.1384,
1650
- "step": 2710
1651
- },
1652
- {
1653
- "epoch": 1.35,
1654
- "learning_rate": 4.655555555555556e-06,
1655
- "loss": 0.1275,
1656
- "step": 2720
1657
- },
1658
- {
1659
- "epoch": 1.35,
1660
- "learning_rate": 4.6416666666666666e-06,
1661
- "loss": 0.1416,
1662
- "step": 2730
1663
- },
1664
- {
1665
- "epoch": 1.36,
1666
- "learning_rate": 4.627777777777778e-06,
1667
- "loss": 0.1119,
1668
- "step": 2740
1669
- },
1670
- {
1671
- "epoch": 1.36,
1672
- "learning_rate": 4.6138888888888886e-06,
1673
- "loss": 0.1079,
1674
- "step": 2750
1675
- },
1676
- {
1677
- "epoch": 1.37,
1678
- "learning_rate": 4.599999999999999e-06,
1679
- "loss": 0.1197,
1680
- "step": 2760
1681
- },
1682
- {
1683
- "epoch": 1.37,
1684
- "learning_rate": 4.5861111111111114e-06,
1685
- "loss": 0.1115,
1686
- "step": 2770
1687
- },
1688
- {
1689
- "epoch": 1.38,
1690
- "learning_rate": 4.572222222222222e-06,
1691
- "loss": 0.1023,
1692
- "step": 2780
1693
- },
1694
- {
1695
- "epoch": 1.38,
1696
- "learning_rate": 4.5583333333333335e-06,
1697
- "loss": 0.1109,
1698
- "step": 2790
1699
- },
1700
- {
1701
- "epoch": 1.39,
1702
- "learning_rate": 4.544444444444444e-06,
1703
- "loss": 0.1184,
1704
- "step": 2800
1705
- },
1706
- {
1707
- "epoch": 1.39,
1708
- "learning_rate": 4.5305555555555555e-06,
1709
- "loss": 0.1103,
1710
- "step": 2810
1711
- },
1712
- {
1713
- "epoch": 1.4,
1714
- "learning_rate": 4.516666666666666e-06,
1715
- "loss": 0.1074,
1716
- "step": 2820
1717
- },
1718
- {
1719
- "epoch": 1.4,
1720
- "learning_rate": 4.502777777777778e-06,
1721
- "loss": 0.1139,
1722
- "step": 2830
1723
- },
1724
- {
1725
- "epoch": 1.4,
1726
- "learning_rate": 4.488888888888889e-06,
1727
- "loss": 0.1107,
1728
- "step": 2840
1729
- },
1730
- {
1731
- "epoch": 1.41,
1732
- "learning_rate": 4.4749999999999995e-06,
1733
- "loss": 0.0777,
1734
- "step": 2850
1735
- },
1736
- {
1737
- "epoch": 1.41,
1738
- "learning_rate": 4.461111111111111e-06,
1739
- "loss": 0.1348,
1740
- "step": 2860
1741
- },
1742
- {
1743
- "epoch": 1.42,
1744
- "learning_rate": 4.4472222222222215e-06,
1745
- "loss": 0.1362,
1746
- "step": 2870
1747
- },
1748
- {
1749
- "epoch": 1.42,
1750
- "learning_rate": 4.433333333333333e-06,
1751
- "loss": 0.1374,
1752
- "step": 2880
1753
- },
1754
- {
1755
- "epoch": 1.43,
1756
- "learning_rate": 4.419444444444444e-06,
1757
- "loss": 0.1135,
1758
- "step": 2890
1759
- },
1760
- {
1761
- "epoch": 1.43,
1762
- "learning_rate": 4.405555555555556e-06,
1763
- "loss": 0.1166,
1764
- "step": 2900
1765
- },
1766
- {
1767
- "epoch": 1.44,
1768
- "learning_rate": 4.391666666666666e-06,
1769
- "loss": 0.1202,
1770
- "step": 2910
1771
- },
1772
- {
1773
- "epoch": 1.44,
1774
- "learning_rate": 4.377777777777778e-06,
1775
- "loss": 0.1455,
1776
- "step": 2920
1777
- },
1778
- {
1779
- "epoch": 1.45,
1780
- "learning_rate": 4.3638888888888884e-06,
1781
- "loss": 0.0831,
1782
- "step": 2930
1783
- },
1784
- {
1785
- "epoch": 1.45,
1786
- "learning_rate": 4.35e-06,
1787
- "loss": 0.1211,
1788
- "step": 2940
1789
- },
1790
- {
1791
- "epoch": 1.46,
1792
- "learning_rate": 4.336111111111111e-06,
1793
- "loss": 0.1194,
1794
- "step": 2950
1795
- },
1796
- {
1797
- "epoch": 1.46,
1798
- "learning_rate": 4.322222222222222e-06,
1799
- "loss": 0.1124,
1800
- "step": 2960
1801
- },
1802
- {
1803
- "epoch": 1.47,
1804
- "learning_rate": 4.308333333333333e-06,
1805
- "loss": 0.0947,
1806
- "step": 2970
1807
- },
1808
- {
1809
- "epoch": 1.47,
1810
- "learning_rate": 4.294444444444444e-06,
1811
- "loss": 0.1152,
1812
- "step": 2980
1813
- },
1814
- {
1815
- "epoch": 1.48,
1816
- "learning_rate": 4.280555555555555e-06,
1817
- "loss": 0.0858,
1818
- "step": 2990
1819
- },
1820
- {
1821
- "epoch": 1.48,
1822
- "learning_rate": 4.266666666666667e-06,
1823
- "loss": 0.087,
1824
- "step": 3000
1825
- },
1826
- {
1827
- "epoch": 1.49,
1828
- "learning_rate": 4.252777777777778e-06,
1829
- "loss": 0.1237,
1830
- "step": 3010
1831
- },
1832
- {
1833
- "epoch": 1.49,
1834
- "learning_rate": 4.238888888888889e-06,
1835
- "loss": 0.0809,
1836
- "step": 3020
1837
- },
1838
- {
1839
- "epoch": 1.5,
1840
- "learning_rate": 4.224999999999999e-06,
1841
- "loss": 0.1022,
1842
- "step": 3030
1843
- },
1844
- {
1845
- "epoch": 1.5,
1846
- "learning_rate": 4.211111111111111e-06,
1847
- "loss": 0.101,
1848
- "step": 3040
1849
- },
1850
- {
1851
- "epoch": 1.51,
1852
- "learning_rate": 4.197222222222221e-06,
1853
- "loss": 0.1327,
1854
- "step": 3050
1855
- },
1856
- {
1857
- "epoch": 1.51,
1858
- "learning_rate": 4.183333333333334e-06,
1859
- "loss": 0.1056,
1860
- "step": 3060
1861
- },
1862
- {
1863
- "epoch": 1.52,
1864
- "learning_rate": 4.169444444444444e-06,
1865
- "loss": 0.1129,
1866
- "step": 3070
1867
- },
1868
- {
1869
- "epoch": 1.52,
1870
- "learning_rate": 4.155555555555556e-06,
1871
- "loss": 0.1119,
1872
- "step": 3080
1873
- },
1874
- {
1875
- "epoch": 1.53,
1876
- "learning_rate": 4.141666666666666e-06,
1877
- "loss": 0.1287,
1878
- "step": 3090
1879
- },
1880
- {
1881
- "epoch": 1.53,
1882
- "learning_rate": 4.127777777777778e-06,
1883
- "loss": 0.1179,
1884
- "step": 3100
1885
- },
1886
- {
1887
- "epoch": 1.54,
1888
- "learning_rate": 4.113888888888889e-06,
1889
- "loss": 0.1152,
1890
- "step": 3110
1891
- },
1892
- {
1893
- "epoch": 1.54,
1894
- "learning_rate": 4.1e-06,
1895
- "loss": 0.0833,
1896
- "step": 3120
1897
- },
1898
- {
1899
- "epoch": 1.55,
1900
- "learning_rate": 4.086111111111111e-06,
1901
- "loss": 0.1304,
1902
- "step": 3130
1903
- },
1904
- {
1905
- "epoch": 1.55,
1906
- "learning_rate": 4.072222222222222e-06,
1907
- "loss": 0.0813,
1908
- "step": 3140
1909
- },
1910
- {
1911
- "epoch": 1.56,
1912
- "learning_rate": 4.058333333333333e-06,
1913
- "loss": 0.1068,
1914
- "step": 3150
1915
- },
1916
- {
1917
- "epoch": 1.56,
1918
- "learning_rate": 4.044444444444444e-06,
1919
- "loss": 0.0998,
1920
- "step": 3160
1921
- },
1922
- {
1923
- "epoch": 1.57,
1924
- "learning_rate": 4.030555555555556e-06,
1925
- "loss": 0.1109,
1926
- "step": 3170
1927
- },
1928
- {
1929
- "epoch": 1.57,
1930
- "learning_rate": 4.016666666666667e-06,
1931
- "loss": 0.1193,
1932
- "step": 3180
1933
- },
1934
- {
1935
- "epoch": 1.58,
1936
- "learning_rate": 4.002777777777778e-06,
1937
- "loss": 0.1234,
1938
- "step": 3190
1939
- },
1940
- {
1941
- "epoch": 1.58,
1942
- "learning_rate": 3.988888888888889e-06,
1943
- "loss": 0.1049,
1944
- "step": 3200
1945
- },
1946
- {
1947
- "epoch": 1.59,
1948
- "learning_rate": 3.975e-06,
1949
- "loss": 0.1017,
1950
- "step": 3210
1951
- },
1952
- {
1953
- "epoch": 1.59,
1954
- "learning_rate": 3.9611111111111115e-06,
1955
- "loss": 0.1044,
1956
- "step": 3220
1957
- },
1958
- {
1959
- "epoch": 1.6,
1960
- "learning_rate": 3.947222222222222e-06,
1961
- "loss": 0.1263,
1962
- "step": 3230
1963
- },
1964
- {
1965
- "epoch": 1.6,
1966
- "learning_rate": 3.933333333333333e-06,
1967
- "loss": 0.1419,
1968
- "step": 3240
1969
- },
1970
- {
1971
- "epoch": 1.61,
1972
- "learning_rate": 3.919444444444444e-06,
1973
- "loss": 0.1294,
1974
- "step": 3250
1975
- },
1976
- {
1977
- "epoch": 1.61,
1978
- "learning_rate": 3.9055555555555555e-06,
1979
- "loss": 0.0939,
1980
- "step": 3260
1981
- },
1982
- {
1983
- "epoch": 1.62,
1984
- "learning_rate": 3.891666666666666e-06,
1985
- "loss": 0.1374,
1986
- "step": 3270
1987
- },
1988
- {
1989
- "epoch": 1.62,
1990
- "learning_rate": 3.8777777777777775e-06,
1991
- "loss": 0.0959,
1992
- "step": 3280
1993
- },
1994
- {
1995
- "epoch": 1.63,
1996
- "learning_rate": 3.863888888888889e-06,
1997
- "loss": 0.1009,
1998
- "step": 3290
1999
- },
2000
- {
2001
- "epoch": 1.63,
2002
- "learning_rate": 3.8499999999999996e-06,
2003
- "loss": 0.1305,
2004
- "step": 3300
2005
- },
2006
- {
2007
- "epoch": 1.64,
2008
- "learning_rate": 3.836111111111111e-06,
2009
- "loss": 0.1303,
2010
- "step": 3310
2011
- },
2012
- {
2013
- "epoch": 1.64,
2014
- "learning_rate": 3.8222222222222224e-06,
2015
- "loss": 0.1282,
2016
- "step": 3320
2017
- },
2018
- {
2019
- "epoch": 1.65,
2020
- "learning_rate": 3.808333333333333e-06,
2021
- "loss": 0.1053,
2022
- "step": 3330
2023
- },
2024
- {
2025
- "epoch": 1.65,
2026
- "learning_rate": 3.794444444444444e-06,
2027
- "loss": 0.1042,
2028
- "step": 3340
2029
- },
2030
- {
2031
- "epoch": 1.66,
2032
- "learning_rate": 3.7805555555555555e-06,
2033
- "loss": 0.0849,
2034
- "step": 3350
2035
- },
2036
- {
2037
- "epoch": 1.66,
2038
- "learning_rate": 3.7666666666666665e-06,
2039
- "loss": 0.1274,
2040
- "step": 3360
2041
- },
2042
- {
2043
- "epoch": 1.67,
2044
- "learning_rate": 3.7527777777777775e-06,
2045
- "loss": 0.1228,
2046
- "step": 3370
2047
- },
2048
- {
2049
- "epoch": 1.67,
2050
- "learning_rate": 3.738888888888889e-06,
2051
- "loss": 0.1129,
2052
- "step": 3380
2053
- },
2054
- {
2055
- "epoch": 1.68,
2056
- "learning_rate": 3.725e-06,
2057
- "loss": 0.1128,
2058
- "step": 3390
2059
- },
2060
- {
2061
- "epoch": 1.68,
2062
- "learning_rate": 3.711111111111111e-06,
2063
- "loss": 0.1317,
2064
- "step": 3400
2065
- },
2066
- {
2067
- "epoch": 1.69,
2068
- "learning_rate": 3.6972222222222224e-06,
2069
- "loss": 0.1246,
2070
- "step": 3410
2071
- },
2072
- {
2073
- "epoch": 1.69,
2074
- "learning_rate": 3.683333333333333e-06,
2075
- "loss": 0.0806,
2076
- "step": 3420
2077
- },
2078
- {
2079
- "epoch": 1.7,
2080
- "learning_rate": 3.669444444444444e-06,
2081
- "loss": 0.119,
2082
- "step": 3430
2083
- },
2084
- {
2085
- "epoch": 1.7,
2086
- "learning_rate": 3.6555555555555554e-06,
2087
- "loss": 0.0893,
2088
- "step": 3440
2089
- },
2090
- {
2091
- "epoch": 1.71,
2092
- "learning_rate": 3.6416666666666664e-06,
2093
- "loss": 0.1058,
2094
- "step": 3450
2095
- },
2096
- {
2097
- "epoch": 1.71,
2098
- "learning_rate": 3.6277777777777774e-06,
2099
- "loss": 0.0944,
2100
- "step": 3460
2101
- },
2102
- {
2103
- "epoch": 1.72,
2104
- "learning_rate": 3.613888888888889e-06,
2105
- "loss": 0.126,
2106
- "step": 3470
2107
- },
2108
- {
2109
- "epoch": 1.72,
2110
- "learning_rate": 3.6e-06,
2111
- "loss": 0.1104,
2112
- "step": 3480
2113
- },
2114
- {
2115
- "epoch": 1.73,
2116
- "learning_rate": 3.5861111111111113e-06,
2117
- "loss": 0.1108,
2118
- "step": 3490
2119
- },
2120
- {
2121
- "epoch": 1.73,
2122
- "learning_rate": 3.5722222222222223e-06,
2123
- "loss": 0.1197,
2124
- "step": 3500
2125
- },
2126
- {
2127
- "epoch": 1.74,
2128
- "learning_rate": 3.558333333333333e-06,
2129
- "loss": 0.1182,
2130
- "step": 3510
2131
- },
2132
- {
2133
- "epoch": 1.74,
2134
- "learning_rate": 3.5444444444444443e-06,
2135
- "loss": 0.0988,
2136
- "step": 3520
2137
- },
2138
- {
2139
- "epoch": 1.75,
2140
- "learning_rate": 3.5305555555555553e-06,
2141
- "loss": 0.0949,
2142
- "step": 3530
2143
- },
2144
- {
2145
- "epoch": 1.75,
2146
- "learning_rate": 3.5166666666666663e-06,
2147
- "loss": 0.1102,
2148
- "step": 3540
2149
- },
2150
- {
2151
- "epoch": 1.76,
2152
- "learning_rate": 3.5027777777777777e-06,
2153
- "loss": 0.1203,
2154
- "step": 3550
2155
- },
2156
- {
2157
- "epoch": 1.76,
2158
- "learning_rate": 3.4888888888888888e-06,
2159
- "loss": 0.1343,
2160
- "step": 3560
2161
- },
2162
- {
2163
- "epoch": 1.77,
2164
- "learning_rate": 3.4749999999999998e-06,
2165
- "loss": 0.0851,
2166
- "step": 3570
2167
- },
2168
- {
2169
- "epoch": 1.77,
2170
- "learning_rate": 3.461111111111111e-06,
2171
- "loss": 0.1181,
2172
- "step": 3580
2173
- },
2174
- {
2175
- "epoch": 1.78,
2176
- "learning_rate": 3.447222222222222e-06,
2177
- "loss": 0.0931,
2178
- "step": 3590
2179
- },
2180
- {
2181
- "epoch": 1.78,
2182
- "learning_rate": 3.433333333333333e-06,
2183
- "loss": 0.1068,
2184
- "step": 3600
2185
- },
2186
- {
2187
- "epoch": 1.79,
2188
- "learning_rate": 3.4194444444444442e-06,
2189
- "loss": 0.0884,
2190
- "step": 3610
2191
- },
2192
- {
2193
- "epoch": 1.79,
2194
- "learning_rate": 3.4055555555555552e-06,
2195
- "loss": 0.1326,
2196
- "step": 3620
2197
- },
2198
- {
2199
- "epoch": 1.8,
2200
- "learning_rate": 3.3916666666666667e-06,
2201
- "loss": 0.1307,
2202
- "step": 3630
2203
- },
2204
- {
2205
- "epoch": 1.8,
2206
- "eval_loss": 0.249267578125,
2207
- "eval_runtime": 590.255,
2208
- "eval_samples_per_second": 40.584,
2209
- "eval_steps_per_second": 5.074,
2210
- "step": 3639
2211
- }
2212
- ],
2213
- "max_steps": 6063,
2214
- "num_train_epochs": 3,
2215
- "total_flos": 134441240494080.0,
2216
- "trial_name": null,
2217
- "trial_params": null
2218
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
training_args.bin DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:c42f106339a0328c004e4952494046976bb21234b5589de38804b01f680386c0
3
- size 5115
 
 
 
 
zero_to_fp32.py DELETED
@@ -1,578 +0,0 @@
1
- #!/usr/bin/env python
2
-
3
- # Copyright (c) Microsoft Corporation.
4
- # SPDX-License-Identifier: Apache-2.0
5
-
6
- # DeepSpeed Team
7
-
8
- # This script extracts fp32 consolidated weights from a zero 2 and 3 DeepSpeed checkpoints. It gets
9
- # copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
10
- # the future. Once extracted, the weights don't require DeepSpeed and can be used in any
11
- # application.
12
- #
13
- # example: python zero_to_fp32.py . pytorch_model.bin
14
-
15
- import argparse
16
- import torch
17
- import glob
18
- import math
19
- import os
20
- import re
21
- from collections import OrderedDict
22
- from dataclasses import dataclass
23
-
24
- # while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
25
- # DeepSpeed data structures it has to be available in the current python environment.
26
- from deepspeed.utils import logger
27
- from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
28
- FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
29
- FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
30
-
31
-
32
- @dataclass
33
- class zero_model_state:
34
- buffers: dict()
35
- param_shapes: dict()
36
- shared_params: list
37
- ds_version: int
38
- frozen_param_shapes: dict()
39
- frozen_param_fragments: dict()
40
-
41
-
42
- debug = 0
43
-
44
- # load to cpu
45
- device = torch.device('cpu')
46
-
47
-
48
- def atoi(text):
49
- return int(text) if text.isdigit() else text
50
-
51
-
52
- def natural_keys(text):
53
- '''
54
- alist.sort(key=natural_keys) sorts in human order
55
- http://nedbatchelder.com/blog/200712/human_sorting.html
56
- (See Toothy's implementation in the comments)
57
- '''
58
- return [atoi(c) for c in re.split(r'(\d+)', text)]
59
-
60
-
61
- def get_model_state_file(checkpoint_dir, zero_stage):
62
- if not os.path.isdir(checkpoint_dir):
63
- raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
64
-
65
- # there should be only one file
66
- if zero_stage == 2:
67
- file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
68
- elif zero_stage == 3:
69
- file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
70
-
71
- if not os.path.exists(file):
72
- raise FileNotFoundError(f"can't find model states file at '{file}'")
73
-
74
- return file
75
-
76
-
77
- def get_checkpoint_files(checkpoint_dir, glob_pattern):
78
- # XXX: need to test that this simple glob rule works for multi-node setup too
79
- ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
80
-
81
- if len(ckpt_files) == 0:
82
- raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
83
-
84
- return ckpt_files
85
-
86
-
87
- def get_optim_files(checkpoint_dir):
88
- return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
89
-
90
-
91
- def get_model_state_files(checkpoint_dir):
92
- return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
93
-
94
-
95
- def parse_model_states(files):
96
- zero_model_states = []
97
- for file in files:
98
- state_dict = torch.load(file, map_location=device)
99
-
100
- if BUFFER_NAMES not in state_dict:
101
- raise ValueError(f"{file} is not a model state checkpoint")
102
- buffer_names = state_dict[BUFFER_NAMES]
103
- if debug:
104
- print("Found buffers:", buffer_names)
105
-
106
- # recover just the buffers while restoring them to fp32 if they were saved in fp16
107
- buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
108
- param_shapes = state_dict[PARAM_SHAPES]
109
-
110
- # collect parameters that are included in param_shapes
111
- param_names = []
112
- for s in param_shapes:
113
- for name in s.keys():
114
- param_names.append(name)
115
-
116
- # update with frozen parameters
117
- frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
118
- if frozen_param_shapes is not None:
119
- if debug:
120
- print(f"Found frozen_param_shapes: {frozen_param_shapes}")
121
- param_names += list(frozen_param_shapes.keys())
122
-
123
- # handle shared params
124
- shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
125
-
126
- ds_version = state_dict.get(DS_VERSION, None)
127
-
128
- frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
129
-
130
- z_model_state = zero_model_state(buffers=buffers,
131
- param_shapes=param_shapes,
132
- shared_params=shared_params,
133
- ds_version=ds_version,
134
- frozen_param_shapes=frozen_param_shapes,
135
- frozen_param_fragments=frozen_param_fragments)
136
- zero_model_states.append(z_model_state)
137
-
138
- return zero_model_states
139
-
140
-
141
- def parse_optim_states(files, ds_checkpoint_dir):
142
-
143
- total_files = len(files)
144
- state_dicts = []
145
- for f in files:
146
- state_dicts.append(torch.load(f, map_location=device))
147
-
148
- if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
149
- raise ValueError(f"{files[0]} is not a zero checkpoint")
150
- zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
151
- world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
152
-
153
- # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
154
- # parameters can be different from data parallelism for non-expert parameters. So we can just
155
- # use the max of the partition_count to get the dp world_size.
156
-
157
- if type(world_size) is list:
158
- world_size = max(world_size)
159
-
160
- if world_size != total_files:
161
- raise ValueError(
162
- f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
163
- "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
164
- )
165
-
166
- # the groups are named differently in each stage
167
- if zero_stage == 2:
168
- fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
169
- elif zero_stage == 3:
170
- fp32_groups_key = FP32_FLAT_GROUPS
171
- else:
172
- raise ValueError(f"unknown zero stage {zero_stage}")
173
-
174
- if zero_stage == 2:
175
- fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
176
- elif zero_stage == 3:
177
- # if there is more than one param group, there will be multiple flattened tensors - one
178
- # flattened tensor per group - for simplicity merge them into a single tensor
179
- #
180
- # XXX: could make the script more memory efficient for when there are multiple groups - it
181
- # will require matching the sub-lists of param_shapes for each param group flattened tensor
182
-
183
- fp32_flat_groups = [
184
- torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
185
- ]
186
-
187
- return zero_stage, world_size, fp32_flat_groups
188
-
189
-
190
- def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir):
191
- """
192
- Returns fp32 state_dict reconstructed from ds checkpoint
193
-
194
- Args:
195
- - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
196
-
197
- """
198
- print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
199
-
200
- optim_files = get_optim_files(ds_checkpoint_dir)
201
- zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
202
- print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
203
-
204
- model_files = get_model_state_files(ds_checkpoint_dir)
205
-
206
- zero_model_states = parse_model_states(model_files)
207
- print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
208
-
209
- if zero_stage == 2:
210
- return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states)
211
- elif zero_stage == 3:
212
- return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states)
213
-
214
-
215
- def _zero2_merge_frozen_params(state_dict, zero_model_states):
216
- if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
217
- return
218
-
219
- frozen_param_shapes = zero_model_states[0].frozen_param_shapes
220
- frozen_param_fragments = zero_model_states[0].frozen_param_fragments
221
-
222
- if debug:
223
- num_elem = sum(s.numel() for s in frozen_param_shapes.values())
224
- print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
225
-
226
- wanted_params = len(frozen_param_shapes)
227
- wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
228
- avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
229
- print(f'Frozen params: Have {avail_numel} numels to process.')
230
- print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
231
-
232
- total_params = 0
233
- total_numel = 0
234
- for name, shape in frozen_param_shapes.items():
235
- total_params += 1
236
- unpartitioned_numel = shape.numel()
237
- total_numel += unpartitioned_numel
238
-
239
- state_dict[name] = frozen_param_fragments[name]
240
-
241
- if debug:
242
- print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
243
-
244
- print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
245
-
246
-
247
- def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
248
- param_shapes = zero_model_states[0].param_shapes
249
-
250
- # Reconstruction protocol:
251
- #
252
- # XXX: document this
253
-
254
- if debug:
255
- for i in range(world_size):
256
- for j in range(len(fp32_flat_groups[0])):
257
- print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
258
-
259
- # XXX: memory usage doubles here (zero2)
260
- num_param_groups = len(fp32_flat_groups[0])
261
- merged_single_partition_of_fp32_groups = []
262
- for i in range(num_param_groups):
263
- merged_partitions = [sd[i] for sd in fp32_flat_groups]
264
- full_single_fp32_vector = torch.cat(merged_partitions, 0)
265
- merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
266
- avail_numel = sum(
267
- [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
268
-
269
- if debug:
270
- wanted_params = sum([len(shapes) for shapes in param_shapes])
271
- wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
272
- # not asserting if there is a mismatch due to possible padding
273
- print(f"Have {avail_numel} numels to process.")
274
- print(f"Need {wanted_numel} numels in {wanted_params} params.")
275
-
276
- # params
277
- # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
278
- # out-of-core computing solution
279
- total_numel = 0
280
- total_params = 0
281
- for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
282
- offset = 0
283
- avail_numel = full_single_fp32_vector.numel()
284
- for name, shape in shapes.items():
285
-
286
- unpartitioned_numel = shape.numel()
287
- total_numel += unpartitioned_numel
288
- total_params += 1
289
-
290
- if debug:
291
- print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
292
- state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
293
- offset += unpartitioned_numel
294
-
295
- # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
296
- # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
297
- # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
298
- # live optimizer object, so we are checking that the numbers are within the right range
299
- align_to = 2 * world_size
300
-
301
- def zero2_align(x):
302
- return align_to * math.ceil(x / align_to)
303
-
304
- if debug:
305
- print(f"original offset={offset}, avail_numel={avail_numel}")
306
-
307
- offset = zero2_align(offset)
308
- avail_numel = zero2_align(avail_numel)
309
-
310
- if debug:
311
- print(f"aligned offset={offset}, avail_numel={avail_numel}")
312
-
313
- # Sanity check
314
- if offset != avail_numel:
315
- raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
316
-
317
- print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
318
-
319
-
320
- def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states):
321
- state_dict = OrderedDict()
322
-
323
- # buffers
324
- buffers = zero_model_states[0].buffers
325
- state_dict.update(buffers)
326
- if debug:
327
- print(f"added {len(buffers)} buffers")
328
-
329
- _zero2_merge_frozen_params(state_dict, zero_model_states)
330
-
331
- _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
332
-
333
- # recover shared parameters
334
- for pair in zero_model_states[0].shared_params:
335
- if pair[1] in state_dict:
336
- state_dict[pair[0]] = state_dict[pair[1]]
337
-
338
- return state_dict
339
-
340
-
341
- def zero3_partitioned_param_info(unpartitioned_numel, world_size):
342
- remainder = unpartitioned_numel % world_size
343
- padding_numel = (world_size - remainder) if remainder else 0
344
- partitioned_numel = math.ceil(unpartitioned_numel / world_size)
345
- return partitioned_numel, padding_numel
346
-
347
-
348
- def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
349
- if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
350
- return
351
-
352
- if debug:
353
- for i in range(world_size):
354
- num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
355
- print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
356
-
357
- frozen_param_shapes = zero_model_states[0].frozen_param_shapes
358
- wanted_params = len(frozen_param_shapes)
359
- wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
360
- avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
361
- print(f'Frozen params: Have {avail_numel} numels to process.')
362
- print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
363
-
364
- total_params = 0
365
- total_numel = 0
366
- for name, shape in zero_model_states[0].frozen_param_shapes.items():
367
- total_params += 1
368
- unpartitioned_numel = shape.numel()
369
- total_numel += unpartitioned_numel
370
-
371
- param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
372
- state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
373
-
374
- partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
375
-
376
- if debug:
377
- print(
378
- f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
379
- )
380
-
381
- print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
382
-
383
-
384
- def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
385
- param_shapes = zero_model_states[0].param_shapes
386
- avail_numel = fp32_flat_groups[0].numel() * world_size
387
- # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
388
- # param, re-consolidating each param, while dealing with padding if any
389
-
390
- # merge list of dicts, preserving order
391
- param_shapes = {k: v for d in param_shapes for k, v in d.items()}
392
-
393
- if debug:
394
- for i in range(world_size):
395
- print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
396
-
397
- wanted_params = len(param_shapes)
398
- wanted_numel = sum(shape.numel() for shape in param_shapes.values())
399
- # not asserting if there is a mismatch due to possible padding
400
- avail_numel = fp32_flat_groups[0].numel() * world_size
401
- print(f"Trainable params: Have {avail_numel} numels to process.")
402
- print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
403
-
404
- # params
405
- # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
406
- # out-of-core computing solution
407
- offset = 0
408
- total_numel = 0
409
- total_params = 0
410
- for name, shape in param_shapes.items():
411
-
412
- unpartitioned_numel = shape.numel()
413
- total_numel += unpartitioned_numel
414
- total_params += 1
415
-
416
- partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
417
-
418
- if debug:
419
- print(
420
- f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
421
- )
422
-
423
- # XXX: memory usage doubles here
424
- state_dict[name] = torch.cat(
425
- tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
426
- 0).narrow(0, 0, unpartitioned_numel).view(shape)
427
- offset += partitioned_numel
428
-
429
- offset *= world_size
430
-
431
- # Sanity check
432
- if offset != avail_numel:
433
- raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
434
-
435
- print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
436
-
437
-
438
- def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states):
439
- state_dict = OrderedDict()
440
-
441
- # buffers
442
- buffers = zero_model_states[0].buffers
443
- state_dict.update(buffers)
444
- if debug:
445
- print(f"added {len(buffers)} buffers")
446
-
447
- _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
448
-
449
- _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
450
-
451
- # recover shared parameters
452
- for pair in zero_model_states[0].shared_params:
453
- if pair[1] in state_dict:
454
- state_dict[pair[0]] = state_dict[pair[1]]
455
-
456
- return state_dict
457
-
458
-
459
- def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None):
460
- """
461
- Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
462
- ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
463
- via a model hub.
464
-
465
- Args:
466
- - ``checkpoint_dir``: path to the desired checkpoint folder
467
- - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
468
-
469
- Returns:
470
- - pytorch ``state_dict``
471
-
472
- Note: this approach may not work if your application doesn't have sufficient free CPU memory and
473
- you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
474
- the checkpoint.
475
-
476
- A typical usage might be ::
477
-
478
- from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
479
- # do the training and checkpoint saving
480
- state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
481
- model = model.cpu() # move to cpu
482
- model.load_state_dict(state_dict)
483
- # submit to model hub or save the model to share with others
484
-
485
- In this example the ``model`` will no longer be usable in the deepspeed context of the same
486
- application. i.e. you will need to re-initialize the deepspeed engine, since
487
- ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
488
-
489
- If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
490
-
491
- """
492
- if tag is None:
493
- latest_path = os.path.join(checkpoint_dir, 'latest')
494
- if os.path.isfile(latest_path):
495
- with open(latest_path, 'r') as fd:
496
- tag = fd.read().strip()
497
- else:
498
- raise ValueError(f"Unable to find 'latest' file at {latest_path}")
499
-
500
- ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
501
-
502
- if not os.path.isdir(ds_checkpoint_dir):
503
- raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
504
-
505
- return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir)
506
-
507
-
508
- def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir, output_file, tag=None):
509
- """
510
- Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
511
- loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
512
-
513
- Args:
514
- - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
515
- - ``output_file``: path to the pytorch fp32 state_dict output file (e.g. path/pytorch_model.bin)
516
- - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
517
- """
518
-
519
- state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
520
- print(f"Saving fp32 state dict to {output_file}")
521
- torch.save(state_dict, output_file)
522
-
523
-
524
- def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
525
- """
526
- 1. Put the provided model to cpu
527
- 2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
528
- 3. Load it into the provided model
529
-
530
- Args:
531
- - ``model``: the model object to update
532
- - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
533
- - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
534
-
535
- Returns:
536
- - ``model`: modified model
537
-
538
- Make sure you have plenty of CPU memory available before you call this function. If you don't
539
- have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
540
- conveniently placed for you in the checkpoint folder.
541
-
542
- A typical usage might be ::
543
-
544
- from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
545
- model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
546
- # submit to model hub or save the model to share with others
547
-
548
- Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
549
- of the same application. i.e. you will need to re-initialize the deepspeed engine, since
550
- ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
551
-
552
- """
553
- logger.info(f"Extracting fp32 weights")
554
- state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
555
-
556
- logger.info(f"Overwriting model with fp32 weights")
557
- model = model.cpu()
558
- model.load_state_dict(state_dict, strict=False)
559
-
560
- return model
561
-
562
-
563
- if __name__ == "__main__":
564
-
565
- parser = argparse.ArgumentParser()
566
- parser.add_argument("checkpoint_dir",
567
- type=str,
568
- help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
569
- parser.add_argument(
570
- "output_file",
571
- type=str,
572
- help="path to the pytorch fp32 state_dict output file (e.g. path/checkpoint-12/pytorch_model.bin)")
573
- parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
574
- args = parser.parse_args()
575
-
576
- debug = args.debug
577
-
578
- convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir, args.output_file)