new data

logs/main_log.txt

@@ -111224,3 +111224,184 @@ saving checkpoint at iteration    6300 to /gpfsscratch/rech/six/commun/checkpoin
   successfully saved checkpoint at iteration    6300 to /gpfsscratch/rech/six/commun/checkpoints/tr8-104B/checkpoints
 time (ms) | save-checkpoint: 18667.92
 [2021-09-30 04:42:10] PULSE: tr8-104B is running for 50:03 since 2021-09-30T03:52:07 (1289770 on 'gpu_p13' partition (r6i4n[5-6,8],r6i5n[4-5],r7i0n[5-8],r7i1n0,r8i2n8,r8i4n1,r8i7n[3-8],r9i0n[0-8],r9i1n[0-8],r9i2n[3-8],r9i3n[7-8],r9i4n[0-2],r9i5n[2,5-7],r9i6n[2-8],r14i7n[1-6])
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.963852
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.090832
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.142243
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.678945
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.864251
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.828453
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.902679
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.871111
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.992544
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.127072
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.985620
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.984376
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.891238
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.953068
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.139466
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.934822
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.054483
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.152760
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.089548
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.032265
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.831478
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.931277
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.114704
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.139945
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.838871
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.793026
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.184644
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.141780
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.185417
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.021588
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.120823
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.081177
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.825079
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.872282
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.952198
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.864201
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.038919
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:11.055901
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.983584
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:43:21 CEST)" was missed by 0:00:10.852635
+ iteration     6310/  159576 | consumed samples:       201600 | elapsed time per iteration (ms): 31055.8 | learning rate: 5.578E-05 | global batch size:    80 | lm loss: 6.324059E+00 | loss scale: 4096.0 | grad norm: 124591.778 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.738472
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.916856
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.453527
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.767151
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.901695
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.960038
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.760247
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.759012
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.796167
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.665853
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.914081
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.709466
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.864119
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.806929
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.865490
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.705922
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.889311
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.914550
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.638906
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.567640
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.603094
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.677329
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.959238
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.916405
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.727715
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.830442
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.813480
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.829148
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.927410
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.599690
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.646915
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.606165
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.726887
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.613497
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.627127
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.638827
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.645783
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.895467
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.758118
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:44:21 CEST)" was missed by 0:00:09.855861
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.645084
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.444049
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.350932
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.423550
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.601918
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.138600
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.288152
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.362393
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.601444
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.452243
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.586762
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.481262
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.515485
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.599178
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.394550
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.514136
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.540861
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.612416
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.549186
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.491972
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.550552
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.284746
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.291169
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.391012
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.574386
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.599623
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.323987
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.298543
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.252696
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.323868
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.644302
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.445363
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.412804
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.580514
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.498539
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.331965
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.411902
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.312190
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.330874
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:45:21 CEST)" was missed by 0:00:07.443200
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.041857
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.578500
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.026635
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.085043
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.885229
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.883961
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.790857
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.863493
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.931870
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.990495
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.014268
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.039506
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.692615
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.728070
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.802336
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.084205
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.041371
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.892144
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.921173
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.852736
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.955440
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.938470
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.039078
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.834459
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.954112
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.980806
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.052355
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.989123
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.724696
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.731123
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.830931
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.763914
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.738478
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.770788
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.883124
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.771905
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.851850
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.752137
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:05.763825
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:46:21 CEST)" was missed by 0:00:06.020483
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.102876
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.064855
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.092455
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.134795
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.077252
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.135608
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.089670
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.004667
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.039735
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.041067
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.090114
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.091998
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.006044
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.031397
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:47:21 CEST)" was missed by 0:00:03.071041
+ iteration     6320/  159576 | consumed samples:       202400 | elapsed time per iteration (ms): 28833.6 | learning rate: 5.600E-05 | global batch size:    80 | lm loss: 6.299813E+00 | loss scale: 4096.0 | grad norm: 122818.261 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     6330/  159576 | consumed samples:       203200 | elapsed time per iteration (ms): 29174.3 | learning rate: 5.622E-05 | global batch size:    80 | lm loss: 6.322478E+00 | loss scale: 4096.0 | grad norm: 120418.031 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)