Hugging Face's logo

bigscience
/
tr8-104B-logs

TensorBoard
Model card Files
xet
Metrics Community
bigscience-bot commited on
Commit
2de7eae
·
1 Parent(s): 2d5b8a6

new data

Browse files
Files changed (1) hide show
  1. logs/main_log.txt +166 -0
logs/main_log.txt CHANGED
@@ -122189,3 +122189,169 @@ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._mea
122189
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:28:21 CEST)" was missed by 0:00:10.436350
122190
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:28:21 CEST)" was missed by 0:00:10.427223
122191
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:28:21 CEST)" was missed by 0:00:10.482622
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
122189
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:28:21 CEST)" was missed by 0:00:10.436350
122190
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:28:21 CEST)" was missed by 0:00:10.427223
122191
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:28:21 CEST)" was missed by 0:00:10.482622
122192
+ iteration 7920/ 159576 | consumed samples: 364400 | elapsed time per iteration (ms): 32264.0 | learning rate: 6.000E-05 | global batch size: 128 | lm loss: 6.883556E+00 | loss scale: 4096.0 | grad norm: 122577.473 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
122193
+ time (ms)
122194
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.491557
122195
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.360635
122196
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.338451
122197
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.397003
122198
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.131162
122199
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.433216
122200
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.291777
122201
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.290542
122202
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.270027
122203
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.445610
122204
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.458872
122205
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.178400
122206
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.448474
122207
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.099218
122208
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.208917
122209
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.170296
122210
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.177295
122211
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.490811
122212
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.327776
122213
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.197480
122214
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.241050
122215
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.387334
122216
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.137689
122217
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.237520
122218
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.446125
122219
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.134698
122220
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.448026
122221
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.259341
122222
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.426991
122223
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.258420
122224
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.420936
122225
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.170522
122226
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.145085
122227
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.395824
122228
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.298811
122229
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.362171
122230
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.345224
122231
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.158894
122232
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:34:21 CEST)" was missed by 0:00:03.289875
122233
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.691335
122234
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.633001
122235
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.538251
122236
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.596800
122237
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.648265
122238
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.491566
122239
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.490330
122240
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.527500
122241
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.469827
122242
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.645438
122243
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.560448
122244
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.587115
122245
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.658710
122246
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.595500
122247
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.378193
122248
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.645889
122249
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.298988
122250
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.377083
122251
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.690580
122252
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.397260
122253
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.331015
122254
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.337484
122255
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.620705
122256
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.370286
122257
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.334497
122258
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.408732
122259
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.370122
122260
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.459112
122261
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.561805
122262
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.544842
122263
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.440856
122264
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.437329
122265
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.458197
122266
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.344852
122267
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.647805
122268
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.498590
122269
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.626787
122270
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.489485
122271
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.185016
122272
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:35:21 CEST)" was missed by 0:00:07.358491
122273
+ iteration 7930/ 159576 | consumed samples: 365680 | elapsed time per iteration (ms): 32012.4 | learning rate: 6.000E-05 | global batch size: 128 | lm loss: 6.774232E+00 | loss scale: 4096.0 | grad norm: 132241.887 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
122274
+ time (ms)
122275
+ iteration 7940/ 159576 | consumed samples: 366960 | elapsed time per iteration (ms): 31069.5 | learning rate: 6.000E-05 | global batch size: 128 | lm loss: 6.763776E+00 | loss scale: 4096.0 | grad norm: 212223.272 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
122276
+ time (ms)
122277
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.748492
122278
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.605789
122279
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.653768
122280
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.300463
122281
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.414489
122282
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.760956
122283
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.446530
122284
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.493753
122285
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.763850
122286
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.485628
122287
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.676041
122288
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.711036
122289
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.736196
122290
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.761441
122291
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.524273
122292
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.607176
122293
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.774241
122294
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.712399
122295
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.806121
122296
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.807014
122297
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.585479
122298
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.677368
122299
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.742319
122300
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.556403
122301
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.453058
122302
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.552865
122303
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.460420
122304
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.450066
122305
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.492709
122306
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.763340
122307
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.643174
122308
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.660430
122309
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.702721
122310
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.573799
122311
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.485895
122312
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.512909
122313
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.605057
122314
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.474084
122315
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.614216
122316
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:45:21 CEST)" was missed by 0:00:03.574741
122317
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.987070
122318
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.949477
122319
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.892313
122320
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.653038
122321
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.844385
122322
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.915800
122323
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.898837
122324
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.999518
122325
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.914547
122326
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:05.012738
122327
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.685092
122328
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.732260
122329
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.999971
122330
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.539023
122331
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.724153
122332
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:05.044678
122333
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:05.045526
122334
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.845730
122335
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.881661
122336
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.823981
122337
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.843486
122338
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.941234
122339
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.950951
122340
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.974784
122341
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:05.002399
122342
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.698915
122343
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.688576
122344
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.712509
122345
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.762824
122346
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.731199
122347
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:05.001882
122348
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.852662
122349
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.751413
122350
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.980850
122351
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.794950
122352
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.691598
122353
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.791412
122354
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.812286
122355
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.724431
122356
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 18:46:21 CEST)" was missed by 0:00:04.813256
122357
+ [2021-09-30 18:45:42] PULSE: tr8-104B is running for 14:53:35 since 2021-09-30T03:52:07 (1289770 on 'gpu_p13' partition (r6i4n[5-6,8],r6i5n[4-5],r7i0n[5-8],r7i1n0,r8i2n8,r8i4n1,r8i7n[3-8],r9i0n[0-8],r9i1n[0-8],r9i2n[3-8],r9i3n[7-8],r9i4n[0-2],r9i5n[2,5-7],r9i6n[2-8],r14i7n[1-6])