| Reading metadata...: 2165it [00:00, 13406.33it/s] | 0/60000 [00:00<?, ?it/s] | |
| Reading metadata...: 1650it [00:00, 9408.60it/s] | |
| [INFO|trainer_utils.py:759] 2023-11-18 13:22:20,012 >> The following columns in the training set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| [WARNING|logging.py:329] 2023-11-18 13:22:22,469 >> `use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`... | |
| 0%| | 19/60000 [05:07<155:59:18, 9.36s/it] | |
| 0%|▏ | 39/60000 [08:15<153:08:24, 9.19s/it] | |
| 0%|▏ | 60/60000 [11:40<168:02:20, 10.09s/it] | |
| 0%|▎ | 79/60000 [14:38<156:59:42, 9.43s/it] | |
| 0%|▍ | 99/60000 [17:51<158:12:45, 9.51s/it] | |
| 0%|▍ | 119/60000 [20:58<160:17:47, 9.64s/it] | |
| 0%|▌ | 139/60000 [24:15<168:07:01, 10.11s/it] | |
| 0%|▋ | 159/60000 [27:26<152:49:26, 9.19s/it] | |
| 0%|▋ | 180/60000 [30:41<151:22:20, 9.11s/it] | |
| 0%|▊ | 199/60000 [33:40<155:35:39, 9.37s/it] | |
| 0%|▉ | 219/60000 [36:46<153:48:42, 9.26s/it] | |
| 0%|▉ | 239/60000 [39:58<159:37:56, 9.62s/it] | |
| 0%|█ | 260/60000 [43:12<156:36:35, 9.44s/it] | |
| 0%|█ | 280/60000 [46:21<156:27:51, 9.43s/it] | |
| 0%|█▏ | 300/60000 [49:37<160:26:51, 9.68s/it] | |
| 1%|█▎ | 319/60000 [53:03<178:10:49, 10.75s/it] | |
| 1%|█▎ | 339/60000 [56:33<172:08:57, 10.39s/it] | |
| 1%|█▍ | 360/60000 [1:00:06<178:39:30, 10.78s/it] | |
| 1%|█▌ | 380/60000 [1:03:34<177:35:38, 10.72s/it] | |
| 1%|█▌ | 399/60000 [1:06:44<164:55:52, 9.96s/it] | |
| 1%|█▋ | 419/60000 [1:10:10<174:01:58, 10.52s/it] | |
| 1%|█▋ | 439/60000 [1:13:28<160:45:30, 9.72s/it] | |
| 1%|█▊ | 460/60000 [1:17:09<176:40:01, 10.68s/it] | |
| 1%|█▉ | 479/60000 [1:20:27<178:05:54, 10.77s/it] | |
| 1%|█▉ | 500/60000 [1:24:13<186:42:12, 11.30s/it] | |
| 1%|██ | 520/60000 [1:27:39<175:12:37, 10.60s/it] | |
| 1%|██▏ | 539/60000 [1:30:54<173:25:57, 10.50s/it] | |
| 1%|██▏ | 559/60000 [1:34:19<178:21:22, 10.80s/it] | |
| 1%|██▎ | 579/60000 [1:37:47<155:43:11, 9.43s/it] | |
| 1%|██▎ | 584/60000 [1:38:34<152:00:31, 9.21s/it] | |
| 1%|██▍ | 599/60000 [1:41:09<158:21:24, 9.60s/it] | |
| 1%|██▍ | 620/60000 [1:44:34<158:26:57, 9.61s/it] | |
| 1%|██▌ | 639/60000 [1:47:59<176:00:16, 10.67s/it] | |
| 1%|██▌ | 660/60000 [1:51:19<150:39:44, 9.14s/it] | |
| 1%|██▋ | 679/60000 [1:54:25<169:43:56, 10.30s/it] | |
| 1%|██▊ | 699/60000 [1:57:40<155:39:16, 9.45s/it] | |
| 1%|██▊ | 719/60000 [2:01:28<156:47:40, 9.52s/it] | |
| 1%|██▉ | 740/60000 [2:05:13<161:48:40, 9.83s/it] | |
| 1%|███ | 759/60000 [2:08:08<150:37:14, 9.15s/it] | |
| 1%|███ | 780/60000 [2:11:29<150:22:36, 9.14s/it] | |
| 1%|███▏ | 800/60000 [2:14:50<175:08:16, 10.65s/it] | |
| 1%|███▎ | 820/60000 [2:17:55<150:59:12, 9.18s/it] | |
| 1%|███▎ | 840/60000 [2:21:03<153:55:45, 9.37s/it] | |
| 1%|███▍ | 860/60000 [2:24:21<159:59:09, 9.74s/it] | |
| 1%|███▍ | 880/60000 [2:28:16<214:56:55, 13.09s/it] | |
| 2%|███▌ | 900/60000 [2:31:23<155:12:49, 9.45s/it] | |
| 2%|███▋ | 920/60000 [2:34:51<236:39:18, 14.42s/it] | |
| 2%|███▋ | 940/60000 [2:38:02<176:39:49, 10.77s/it] | |
| Reading metadata...: 1650it [00:00, 9319.53it/s] | 959/60000 [2:41:02<160:42:05, 9.80s/it] | |
| Reading metadata...: 1it [00:00, 5.88it/s] | |
| 2%|███▉ | 979/60000 [2:44:20<150:57:53, 9.21s/it] | |
| 2%|███▉ | 999/60000 [2:47:26<150:25:55, 9.18s/it] | |
| 2%|███▉ | 1000/60000 [2:47:35<150:19:28, 9.17s/it][INFO|trainer.py:3173] 2023-11-18 16:07:59,097 >> ***** Running Evaluation ***** | |
| [INFO|trainer.py:3177] 2023-11-18 16:07:59,098 >> Num examples: Unknown | |
| [INFO|trainer.py:3178] 2023-11-18 16:07:59,098 >> Batch size = 4 | |
| [INFO|trainer_utils.py:759] 2023-11-18 16:08:03,223 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale. If input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| {'eval_loss': 0.129150390625, 'eval_wer': 9.498680738786279, 'eval_runtime': 612.1544, 'eval_samples_per_second': 2.784, 'eval_steps_per_second': 0.696, 'epoch': 0.02} | |
| 2%|███▉ | 1000/60000 [2:57:47<150:19:28, 9.17s/it][INFO|trainer.py:2896] 2023-11-18 16:18:12,872 >> Saving model checkpoint to ./checkpoint-1000 | |
| [INFO|configuration_utils.py:462] 2023-11-18 16:18:12,881 >> Configuration saved in ./checkpoint-1000/config.json | |
| [INFO|configuration_utils.py:568] 2023-11-18 16:18:12,893 >> Configuration saved in ./checkpoint-1000/generation_config.json | |
| [2023-11-18 16:18:42,519] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step1000 is about to be saved! | |
| [INFO|modeling_utils.py:2194] 2023-11-18 16:18:42,280 >> Model weights saved in ./checkpoint-1000/pytorch_model.bin | |
| [INFO|feature_extraction_utils.py:425] 2023-11-18 16:18:42,309 >> Feature extractor saved in ./checkpoint-1000/preprocessor_config.json | |
| [2023-11-18 16:18:42,846] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-1000/global_step1000/mp_rank_00_model_states.pt | |
| [2023-11-18 16:18:42,846] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-1000/global_step1000/mp_rank_00_model_states.pt... | |
| [2023-11-18 16:19:53,693] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-1000/global_step1000/mp_rank_00_model_states.pt. | |
| [2023-11-18 16:19:56,191] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-1000/global_step1000/zero_pp_rank_0_mp_rank_00_optim_states.pt... | |
| [2023-11-18 16:20:15,747] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-1000/global_step1000/zero_pp_rank_0_mp_rank_00_optim_states.pt. | |
| [2023-11-18 16:20:15,827] [INFO] [engine.py:3417:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-1000/global_step1000/zero_pp_rank_0_mp_rank_00_optim_states.pt | |
| [2023-11-18 16:20:15,831] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step1000 is ready now! | |
| [INFO|feature_extraction_utils.py:425] 2023-11-18 16:21:11,847 >> Feature extractor saved in ./preprocessor_config.json | |
| 2%|████ | 1019/60000 [3:03:56<160:03:33, 9.77s/it] | |
| 2%|████ | 1040/60000 [3:07:24<154:19:06, 9.42s/it] | |
| 2%|████▏ | 1060/60000 [3:10:30<150:30:16, 9.19s/it] | |
| 2%|████▎ | 1079/60000 [3:13:36<160:28:49, 9.81s/it] | |
| 2%|████▎ | 1099/60000 [3:16:51<154:50:54, 9.46s/it] | |
| 2%|████▍ | 1119/60000 [3:20:07<163:46:21, 10.01s/it] | |
| 2%|████▍ | 1139/60000 [3:23:25<161:44:53, 9.89s/it] | |
| 2%|████▌ | 1160/60000 [3:26:43<151:47:46, 9.29s/it] | |
| 2%|████▋ | 1180/60000 [3:29:56<157:42:37, 9.65s/it] | |
| 2%|████▋ | 1199/60000 [3:32:53<156:43:48, 9.60s/it] | |
| 2%|████▊ | 1220/60000 [3:36:19<153:50:38, 9.42s/it] | |
| 2%|████▉ | 1240/60000 [3:39:40<162:05:41, 9.93s/it] | |
| 2%|████▉ | 1259/60000 [3:42:40<154:56:28, 9.50s/it] | |
| 2%|█████ | 1268/60000 [3:44:08<153:51:32, 9.43s/it] | |
| 2%|█████ | 1279/60000 [3:45:58<159:14:10, 9.76s/it] | |
| 2%|█████▏ | 1299/60000 [3:49:18<169:32:55, 10.40s/it] | |
| 2%|█████▏ | 1320/60000 [3:52:53<160:35:35, 9.85s/it] | |
| 2%|█████▎ | 1339/60000 [3:56:03<151:16:59, 9.28s/it] | |
| 2%|█████▎ | 1360/60000 [3:59:18<150:14:39, 9.22s/it] | |
| 2%|█████▍ | 1379/60000 [4:02:24<161:24:32, 9.91s/it] | |
| 2%|█████▌ | 1399/60000 [4:05:36<150:50:03, 9.27s/it] | |
| 2%|█████▌ | 1420/60000 [4:08:58<153:51:31, 9.46s/it] | |
| 2%|█████▋ | 1440/60000 [4:12:32<268:42:47, 16.52s/it] | |
| 2%|█████▊ | 1459/60000 [4:15:40<167:25:48, 10.30s/it] | |
| 2%|█████▊ | 1479/60000 [4:19:20<180:20:19, 11.09s/it] | |
| 2%|█████▉ | 1499/60000 [4:22:39<158:46:13, 9.77s/it] | |
| 3%|██████ | 1520/60000 [4:26:11<158:51:45, 9.78s/it] | |
| 3%|██████ | 1539/60000 [4:29:26<162:36:09, 10.01s/it] | |
| 3%|██████▏ | 1560/60000 [4:33:03<173:55:15, 10.71s/it] | |
| 3%|██████▏ | 1580/60000 [4:36:23<161:38:39, 9.96s/it] | |
| 3%|██████▎ | 1586/60000 [4:37:19<142:22:29, 8.77s/it] | |
| 3%|██████▎ | 1587/60000 [4:37:26<136:23:29, 8.41s/it] | |
| 3%|██████▎ | 1600/60000 [4:39:40<166:18:31, 10.25s/it] | |
| 3%|██████▍ | 1619/60000 [4:42:51<158:19:48, 9.76s/it] | |
| 3%|██████▍ | 1640/60000 [4:46:34<179:40:16, 11.08s/it] | |
| 3%|██████▌ | 1660/60000 [4:49:57<159:16:34, 9.83s/it] | |
| 3%|██████▋ | 1680/60000 [4:53:13<159:55:32, 9.87s/it] | |
| 3%|██████▋ | 1700/60000 [4:56:31<160:07:22, 9.89s/it] | |
| 3%|██████▊ | 1720/60000 [4:59:48<160:30:27, 9.91s/it] | |
| 3%|██████▊ | 1740/60000 [5:03:12<163:35:37, 10.11s/it] | |
| 3%|██████▉ | 1759/60000 [5:06:50<184:16:28, 11.39s/it] | |
| 3%|███████ | 1779/60000 [5:10:08<163:21:51, 10.10s/it] | |
| 3%|███████ | 1799/60000 [5:14:02<163:45:05, 10.13s/it] | |
| 3%|███████▏ | 1819/60000 [5:17:18<159:26:20, 9.87s/it] | |
| 3%|███████▎ | 1839/60000 [5:20:39<159:09:01, 9.85s/it] | |
| 3%|███████▎ | 1859/60000 [5:23:48<153:23:42, 9.50s/it] | |
| 3%|███████▍ | 1879/60000 [5:27:22<159:48:10, 9.90s/it] | |
| 3%|███████▌ | 1900/60000 [5:30:41<150:21:31, 9.32s/it] | |
| 3%|███████▌ | 1919/60000 [5:33:49<156:00:28, 9.67s/it] | |
| 3%|███████▋ | 1940/60000 [5:37:08<153:47:37, 9.54s/it] | |
| Reading metadata...: 1650it [00:00, 7956.90it/s] | 1949/60000 [5:38:34<154:59:37, 9.61s/it] | |
| 3%|███████▋ | 1960/60000 [5:40:18<151:20:44, 9.39s/it] | |
| 3%|███████▊ | 1980/60000 [5:43:36<152:42:22, 9.48s/it] | |
| 3%|███████▉ | 1999/60000 [5:46:40<158:55:24, 9.86s/it] | |
| 3%|███████▉ | 2000/60000 [5:46:54<179:09:00, 11.12s/it][INFO|trainer.py:3173] 2023-11-18 19:07:17,798 >> ***** Running Evaluation ***** | |
| [INFO|trainer.py:3177] 2023-11-18 19:07:17,799 >> Num examples: Unknown | |
| [INFO|trainer.py:3178] 2023-11-18 19:07:17,799 >> Batch size = 4 | |
| Reading metadata...: 1704it [00:00, 9996.31it/s] | |
| [INFO|trainer_utils.py:759] 2023-11-18 19:07:18,883 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale. If input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| 3%|███████▉ | 2000/60000 [5:57:35<179:09:00, 11.12s/it] | |
| 3%|███████▉ | 2000/60000 [5:57:35<179:09:00, 11.12s/it][INFO|trainer.py:2896] 2023-11-18 19:18:28,722 >> Saving model checkpoint to ./checkpoint-2000 | |
| [INFO|configuration_utils.py:462] 2023-11-18 19:18:28,733 >> Configuration saved in ./checkpoint-2000/config.json | |
| [INFO|configuration_utils.py:568] 2023-11-18 19:18:28,740 >> Configuration saved in ./checkpoint-2000/generation_config.json | |
| [INFO|modeling_utils.py:2194] 2023-11-18 19:18:49,012 >> Model weights saved in ./checkpoint-2000/pytorch_model.bin | |
| [INFO|feature_extraction_utils.py:425] 2023-11-18 19:18:49,017 >> Feature extractor saved in ./checkpoint-2000/preprocessor_config.json | |
| [2023-11-18 19:18:49,048] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step2000 is about to be saved! | |
| [2023-11-18 19:18:49,076] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-2000/global_step2000/mp_rank_00_model_states.pt | |
| [2023-11-18 19:18:49,077] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-2000/global_step2000/mp_rank_00_model_states.pt... | |
| [2023-11-18 19:18:58,661] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-2000/global_step2000/mp_rank_00_model_states.pt. | |
| [2023-11-18 19:18:58,678] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-2000/global_step2000/zero_pp_rank_0_mp_rank_00_optim_states.pt... | |
| [2023-11-18 19:19:20,153] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-2000/global_step2000/zero_pp_rank_0_mp_rank_00_optim_states.pt. | |
| [2023-11-18 19:19:20,168] [INFO] [engine.py:3417:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-2000/global_step2000/zero_pp_rank_0_mp_rank_00_optim_states.pt | |
| [2023-11-18 19:19:20,168] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step2000 is ready now! | |
| [INFO|feature_extraction_utils.py:425] 2023-11-18 19:20:30,717 >> Feature extractor saved in ./preprocessor_config.json | |
| 3%|███████▉ | 2020/60000 [6:03:46<170:10:04, 10.57s/it] | |
| 3%|████████ | 2039/60000 [6:07:12<185:41:14, 11.53s/it] | |
| 3%|████████▏ | 2060/60000 [6:11:16<189:05:09, 11.75s/it] | |
| 3%|████████▏ | 2079/60000 [6:14:48<176:42:21, 10.98s/it] | |
| 4%|████████▎ | 2100/60000 [6:18:45<173:51:35, 10.81s/it] | |
| 4%|████████▎ | 2119/60000 [6:22:10<176:03:40, 10.95s/it] | |
| 4%|████████▍ | 2139/60000 [6:25:45<171:19:42, 10.66s/it] | |
| 4%|████████▌ | 2159/60000 [6:29:23<173:59:19, 10.83s/it] | |
| 4%|████████▌ | 2180/60000 [6:33:41<190:12:54, 11.84s/it] | |
| 4%|████████▋ | 2199/60000 [6:37:08<176:47:44, 11.01s/it] | |
| 4%|████████▊ | 2220/60000 [6:41:00<178:52:03, 11.14s/it] | |
| 4%|████████▊ | 2240/60000 [6:44:47<164:05:45, 10.23s/it] | |
| 4%|████████▉ | 2260/60000 [6:48:02<153:27:48, 9.57s/it] | |
| 4%|█████████ | 2280/60000 [6:51:15<149:25:16, 9.32s/it] | |
| 4%|█████████ | 2300/60000 [6:54:27<154:23:17, 9.63s/it] | |
| 4%|█████████▏ | 2320/60000 [6:58:07<173:37:49, 10.84s/it] | |
| 4%|█████████▏ | 2340/60000 [7:01:34<152:59:29, 9.55s/it] | |
| 4%|█████████▎ | 2360/60000 [7:04:48<154:56:48, 9.68s/it] | |
| 4%|█████████▍ | 2380/60000 [7:07:58<154:08:22, 9.63s/it] | |
| 4%|█████████▍ | 2399/60000 [7:10:58<155:41:52, 9.73s/it] | |
| 4%|█████████▌ | 2419/60000 [7:14:08<153:47:51, 9.62s/it] | |
| 4%|█████████▋ | 2440/60000 [7:17:43<183:45:19, 11.49s/it] | |
| 4%|█████████▋ | 2459/60000 [7:20:58<162:35:26, 10.17s/it] | |
| 4%|█████████▊ | 2480/60000 [7:24:41<173:43:25, 10.87s/it] | |
| 4%|█████████▉ | 2500/60000 [7:28:20<175:05:42, 10.96s/it] | |
| 4%|█████████▉ | 2519/60000 [7:31:49<172:25:52, 10.80s/it] | |
| 4%|██████████ | 2539/60000 [7:35:20<165:39:31, 10.38s/it] | |
| 4%|██████████ | 2559/60000 [7:38:33<148:34:23, 9.31s/it] | |
| Reading metadata...: 2165it [00:00, 13541.35it/s] | 2567/60000 [7:39:52<157:49:06, 9.89s/it] | |
| 4%|██████████▏ | 2579/60000 [7:41:50<156:11:18, 9.79s/it] | |
| 4%|██████████▏ | 2587/60000 [7:43:09<155:56:19, 9.78s/it] | |
| 4%|██████████▏ | 2588/60000 [7:43:16<139:44:36, 8.76s/it] | |
| 4%|██████████▎ | 2599/60000 [7:45:00<161:38:16, 10.14s/it] | |
| 4%|██████████▎ | 2619/60000 [7:48:22<171:08:06, 10.74s/it] | |
| 4%|██████████▍ | 2639/60000 [7:51:47<166:31:19, 10.45s/it] | |
| 4%|██████████▌ | 2659/60000 [7:54:58<146:36:44, 9.20s/it] | |
| 4%|██████████▌ | 2679/60000 [7:58:13<157:55:43, 9.92s/it] | |
| 4%|██████████▋ | 2699/60000 [8:01:59<168:24:12, 10.58s/it] | |
| 5%|██████████▋ | 2719/60000 [8:05:21<157:58:45, 9.93s/it] | |
| 5%|██████████▊ | 2740/60000 [8:08:48<158:54:40, 9.99s/it] | |
| 5%|██████████▉ | 2759/60000 [8:12:03<160:34:58, 10.10s/it] | |
| 5%|██████████▉ | 2780/60000 [8:15:26<153:25:37, 9.65s/it] | |
| 5%|███████████ | 2799/60000 [8:18:38<163:21:25, 10.28s/it] | |
| 5%|███████████▏ | 2819/60000 [8:21:52<150:18:25, 9.46s/it] | |
| 5%|███████████▏ | 2840/60000 [8:25:29<153:24:38, 9.66s/it] | |
| 5%|███████████▎ | 2860/60000 [8:28:37<147:51:10, 9.32s/it] | |
| 5%|███████████▎ | 2879/60000 [8:31:37<150:12:33, 9.47s/it] | |
| 5%|███████████▍ | 2900/60000 [8:35:00<147:54:34, 9.33s/it] | |
| 5%|███████████▌ | 2919/60000 [8:38:30<159:24:06, 10.05s/it] | |
| Reading metadata...: 1650it [00:00, 9387.15it/s] | 2940/60000 [8:41:50<148:31:36, 9.37s/it] | |
| Reading metadata...: 1it [00:00, 6.07it/s] | |
| 5%|███████████▋ | 2960/60000 [8:45:01<146:52:43, 9.27s/it] | |
| 5%|███████████▊ | 2980/60000 [8:48:37<165:54:03, 10.47s/it] | |
| 5%|███████████▊ | 2999/60000 [8:51:34<147:36:55, 9.32s/it] | |
| 5%|███████████▊ | 3000/60000 [8:51:44<148:50:20, 9.40s/it][INFO|trainer.py:3173] 2023-11-18 22:12:08,177 >> ***** Running Evaluation ***** | |
| [INFO|trainer.py:3177] 2023-11-18 22:12:08,177 >> Num examples: Unknown | |
| [INFO|trainer.py:3178] 2023-11-18 22:12:08,177 >> Batch size = 4 | |
| Reading metadata...: 1704it [00:00, 9733.62it/s] | |
| [INFO|trainer_utils.py:759] 2023-11-18 22:12:09,317 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale. If input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| 5%|███████████▊ | 3000/60000 [9:01:57<148:50:20, 9.40s/it] | |
| 5%|███████████▊ | 3000/60000 [9:01:57<148:50:20, 9.40s/it][INFO|trainer.py:2896] 2023-11-18 22:22:43,229 >> Saving model checkpoint to ./checkpoint-3000 | |
| [INFO|configuration_utils.py:462] 2023-11-18 22:22:43,238 >> Configuration saved in ./checkpoint-3000/config.json | |
| [INFO|configuration_utils.py:568] 2023-11-18 22:22:43,253 >> Configuration saved in ./checkpoint-3000/generation_config.json | |
| [INFO|modeling_utils.py:2194] 2023-11-18 22:23:13,532 >> Model weights saved in ./checkpoint-3000/pytorch_model.bin | |
| [INFO|feature_extraction_utils.py:425] 2023-11-18 22:23:13,536 >> Feature extractor saved in ./checkpoint-3000/preprocessor_config.json | |
| [2023-11-18 22:23:13,554] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step3000 is about to be saved! | |
| [2023-11-18 22:23:13,579] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-3000/global_step3000/mp_rank_00_model_states.pt | |
| [2023-11-18 22:23:13,580] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-3000/global_step3000/mp_rank_00_model_states.pt... | |
| [2023-11-18 22:23:35,807] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-3000/global_step3000/mp_rank_00_model_states.pt. | |
| [2023-11-18 22:23:35,835] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-3000/global_step3000/zero_pp_rank_0_mp_rank_00_optim_states.pt... | |
| [2023-11-18 22:24:20,153] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-3000/global_step3000/zero_pp_rank_0_mp_rank_00_optim_states.pt. | |
| [2023-11-18 22:24:20,166] [INFO] [engine.py:3417:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-3000/global_step3000/zero_pp_rank_0_mp_rank_00_optim_states.pt | |
| [2023-11-18 22:24:20,167] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step3000 is ready now! | |
| [INFO|feature_extraction_utils.py:425] 2023-11-18 22:25:27,252 >> Feature extractor saved in ./preprocessor_config.json | |
| 5%|███████████▉ | 3020/60000 [9:08:44<173:24:44, 10.96s/it] | |
| 5%|████████████ | 3039/60000 [9:12:00<157:22:11, 9.95s/it] | |
| 5%|████████████ | 3060/60000 [9:15:33<175:26:09, 11.09s/it] | |
| 5%|████████████▏ | 3080/60000 [9:18:43<147:03:08, 9.30s/it] | |
| 5%|████████████▏ | 3100/60000 [9:21:53<147:43:23, 9.35s/it] | |
| 5%|████████████▎ | 3120/60000 [9:25:01<146:15:26, 9.26s/it] | |
| 5%|████████████▍ | 3140/60000 [9:28:07<148:10:42, 9.38s/it] | |
| 5%|████████████▍ | 3160/60000 [9:31:20<149:41:47, 9.48s/it] | |
| 5%|████████████▌ | 3180/60000 [9:34:26<150:53:25, 9.56s/it] | |
| 5%|████████████▋ | 3200/60000 [9:37:35<152:14:12, 9.65s/it] | |
| 5%|████████████▋ | 3219/60000 [9:40:46<173:48:32, 11.02s/it] | |
| 5%|████████████▊ | 3240/60000 [9:44:13<153:18:48, 9.72s/it] | |
| 5%|████████████▉ | 3260/60000 [9:47:25<151:50:02, 9.63s/it] | |
| 5%|████████████▉ | 3280/60000 [9:50:35<148:13:52, 9.41s/it] | |
| 5%|█████████████ | 3299/60000 [9:53:36<149:55:15, 9.52s/it] | |
| 6%|█████████████ | 3319/60000 [9:57:00<146:51:45, 9.33s/it] | |
| 6%|█████████████▏ | 3339/60000 [10:00:16<155:37:45, 9.89s/it] | |
| 6%|█████████████▏ | 3359/60000 [10:03:23<145:49:06, 9.27s/it] | |
| 6%|█████████████▎ | 3379/60000 [10:06:33<146:18:05, 9.30s/it] | |
| 6%|█████████████▎ | 3399/60000 [10:09:43<149:17:20, 9.50s/it] | |
| 6%|█████████████▍ | 3419/60000 [10:12:57<150:31:35, 9.58s/it] | |
| 6%|█████████████▌ | 3439/60000 [10:16:06<147:46:21, 9.41s/it] | |
| 6%|█████████████▌ | 3459/60000 [10:19:22<160:31:38, 10.22s/it] | |
| 6%|█████████████▋ | 3479/60000 [10:22:37<159:07:42, 10.14s/it] | |
| 6%|█████████████▊ | 3499/60000 [10:26:14<177:49:44, 11.33s/it] | |
| 6%|█████████████▊ | 3519/60000 [10:29:37<159:41:01, 10.18s/it] | |
| 6%|█████████████▉ | 3540/60000 [10:34:02<232:06:45, 14.80s/it] | |
| 6%|██████████████ | 3560/60000 [10:37:20<153:38:38, 9.80s/it] | |
| 6%|██████████████ | 3580/60000 [10:40:40<165:10:30, 10.54s/it] | |
| 6%|██████████████ | 3589/60000 [10:42:15<163:24:00, 10.43s/it] | |
| 6%|██████████████ | 3590/60000 [10:42:21<142:14:41, 9.08s/it] | |
| 6%|██████████████▏ | 3599/60000 [10:44:05<259:50:54, 16.59s/it] | |
| 6%|██████████████▏ | 3620/60000 [10:47:33<146:19:11, 9.34s/it] | |
| 6%|██████████████▎ | 3640/60000 [10:50:59<157:38:13, 10.07s/it] | |
| 6%|██████████████▍ | 3660/60000 [10:54:41<159:33:31, 10.20s/it] | |
| 6%|██████████████▍ | 3680/60000 [10:58:01<152:21:31, 9.74s/it] | |
| 6%|██████████████▌ | 3699/60000 [11:01:13<161:01:11, 10.30s/it] | |
| 6%|██████████████▋ | 3720/60000 [11:04:57<171:00:07, 10.94s/it] | |
| 6%|██████████████▋ | 3739/60000 [11:08:34<169:17:31, 10.83s/it] | |
| 6%|██████████████▊ | 3759/60000 [11:12:05<163:42:03, 10.48s/it] | |
| 6%|██████████████▊ | 3779/60000 [11:15:58<180:54:54, 11.58s/it] | |
| 6%|██████████████▉ | 3799/60000 [11:19:36<167:48:48, 10.75s/it] | |
| 6%|███████████████ | 3820/60000 [11:23:16<160:39:18, 10.29s/it] | |
| 6%|███████████████ | 3840/60000 [11:26:46<162:55:27, 10.44s/it] | |
| 6%|███████████████▏ | 3860/60000 [11:30:18<160:33:33, 10.30s/it] | |
| Reading metadata...: 2165it [00:00, 12148.99it/s] | 3866/60000 [11:31:24<170:58:24, 10.96s/it] | |
| 6%|███████████████▎ | 3880/60000 [11:33:48<152:05:35, 9.76s/it] | |
| 6%|███████████████▎ | 3900/60000 [11:37:02<148:15:10, 9.51s/it] | |
| 7%|███████████████▍ | 3920/60000 [11:40:14<148:00:20, 9.50s/it] | |
| Reading metadata...: 1650it [00:00, 10018.14it/s] | 3930/60000 [11:41:51<151:38:12, 9.74s/it] | |
| 7%|███████████████▍ | 3940/60000 [11:43:38<178:37:43, 11.47s/it] | |
| 7%|███████████████▌ | 3960/60000 [11:47:23<170:53:14, 10.98s/it] | |
| 7%|███████████████▋ | 3980/60000 [11:51:14<171:34:39, 11.03s/it] | |
| 7%|███████████████▋ | 4000/60000 [11:54:48<166:35:19, 10.71s/it][INFO|trainer.py:3173] 2023-11-19 01:15:12,460 >> ***** Running Evaluation ***** | |
| [INFO|trainer.py:3177] 2023-11-19 01:15:12,462 >> Num examples: Unknown | |
| [INFO|trainer.py:3178] 2023-11-19 01:15:12,462 >> Batch size = 4 | |
| Reading metadata...: 1it [00:00, 6.53it/s] | |
| [INFO|trainer_utils.py:759] 2023-11-19 01:15:13,424 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale. If input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| {'eval_loss': 0.1268310546875, 'eval_wer': 8.89558989822842, 'eval_runtime': 729.8005, 'eval_samples_per_second': 2.335, 'eval_steps_per_second': 0.584, 'epoch': 0.07} | |
| 7%|███████████████▋ | 4000/60000 [12:06:58<166:35:19, 10.71s/it][INFO|trainer.py:2896] 2023-11-19 01:27:51,852 >> Saving model checkpoint to ./checkpoint-4000 | |
| [INFO|configuration_utils.py:462] 2023-11-19 01:27:51,865 >> Configuration saved in ./checkpoint-4000/config.json | |
| [INFO|configuration_utils.py:568] 2023-11-19 01:27:51,873 >> Configuration saved in ./checkpoint-4000/generation_config.json | |
| [2023-11-19 01:28:38,504] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step4000 is about to be saved! | |
| [2023-11-19 01:28:38,543] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-4000/global_step4000/mp_rank_00_model_states.pt | |
| [2023-11-19 01:28:38,543] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-4000/global_step4000/mp_rank_00_model_states.pt... | |
| [INFO|modeling_utils.py:2194] 2023-11-19 01:28:38,451 >> Model weights saved in ./checkpoint-4000/pytorch_model.bin | |
| [INFO|feature_extraction_utils.py:425] 2023-11-19 01:28:38,460 >> Feature extractor saved in ./checkpoint-4000/preprocessor_config.json | |
| [2023-11-19 01:28:44,210] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-4000/global_step4000/mp_rank_00_model_states.pt. | |
| [2023-11-19 01:28:44,220] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-4000/global_step4000/zero_pp_rank_0_mp_rank_00_optim_states.pt... | |
| [2023-11-19 01:29:07,634] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-4000/global_step4000/zero_pp_rank_0_mp_rank_00_optim_states.pt. | |
| [2023-11-19 01:29:07,668] [INFO] [engine.py:3417:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-4000/global_step4000/zero_pp_rank_0_mp_rank_00_optim_states.pt | |
| [2023-11-19 01:29:07,668] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step4000 is ready now! | |
| [INFO|feature_extraction_utils.py:425] 2023-11-19 01:30:17,780 >> Feature extractor saved in ./preprocessor_config.json | |
| 7%|███████████████▊ | 4019/60000 [12:13:31<171:24:49, 11.02s/it] | |
| 7%|███████████████▉ | 4039/60000 [12:17:10<164:37:02, 10.59s/it] | |
| 7%|███████████████▉ | 4060/60000 [12:20:57<155:35:39, 10.01s/it] | |
| 7%|████████████████ | 4080/60000 [12:24:15<148:47:04, 9.58s/it] | |
| 7%|████████████████ | 4099/60000 [12:27:10<142:22:51, 9.17s/it] | |
| 7%|████████████████▏ | 4119/60000 [12:30:18<149:47:00, 9.65s/it] | |
| 7%|████████████████▎ | 4139/60000 [12:33:21<142:13:05, 9.17s/it] | |
| 7%|████████████████▎ | 4159/60000 [12:36:24<141:09:26, 9.10s/it] | |
| 7%|████████████████▍ | 4180/60000 [12:39:38<141:22:34, 9.12s/it] | |
| 7%|████████████████▌ | 4199/60000 [12:42:32<143:07:23, 9.23s/it] | |
| 7%|████████████████▌ | 4220/60000 [12:45:49<140:44:27, 9.08s/it] | |
| 7%|████████████████▋ | 4240/60000 [12:48:52<142:41:50, 9.21s/it] | |
| 7%|████████████████▊ | 4260/60000 [12:51:57<141:40:28, 9.15s/it] | |
| 7%|████████████████▊ | 4279/60000 [12:54:51<143:00:35, 9.24s/it] | |
| 7%|████████████████▉ | 4300/60000 [12:58:09<146:41:35, 9.48s/it] | |
| 7%|████████████████▉ | 4319/60000 [13:01:17<143:17:07, 9.26s/it] | |
| 7%|█████████████████ | 4339/60000 [13:04:19<140:12:50, 9.07s/it] | |
| 7%|█████████████████▏ | 4360/60000 [13:07:32<141:53:59, 9.18s/it] | |
| 7%|█████████████████▏ | 4380/60000 [13:10:38<145:43:12, 9.43s/it] | |
| 7%|█████████████████▎ | 4399/60000 [13:14:23<142:17:34, 9.21s/it] | |
| 7%|█████████████████▍ | 4419/60000 [13:17:25<140:58:10, 9.13s/it] | |
| 7%|█████████████████▍ | 4439/60000 [13:20:31<141:51:43, 9.19s/it] | |
| 7%|█████████████████▌ | 4460/60000 [13:23:45<141:09:48, 9.15s/it] | |
| 7%|█████████████████▌ | 4480/60000 [13:27:06<177:08:56, 11.49s/it] | |
| 8%|█████████████████▋ | 4500/60000 [13:30:08<139:35:12, 9.05s/it] | |
| 8%|█████████████████▊ | 4520/60000 [13:33:36<140:19:47, 9.11s/it] | |
| 8%|█████████████████▊ | 4539/60000 [13:36:30<140:32:06, 9.12s/it] | |
| 8%|█████████████████▉ | 4559/60000 [13:39:38<150:37:32, 9.78s/it] | |
| 8%|██████████████████ | 4579/60000 [13:42:45<144:33:59, 9.39s/it] | |
| 8%|██████████████████ | 4592/60000 [13:44:38<123:38:09, 8.03s/it] | |
| 8%|██████████████████ | 4593/60000 [13:44:44<112:47:52, 7.33s/it] | |
| 8%|██████████████████ | 4600/60000 [13:45:48<138:21:37, 8.99s/it] | |
| 8%|██████████████████▏ | 4619/60000 [13:48:41<138:11:57, 8.98s/it] | |
| 8%|██████████████████▏ | 4639/60000 [13:51:53<144:05:49, 9.37s/it] | |
| 8%|██████████████████▎ | 4660/60000 [13:55:10<142:38:56, 9.28s/it] | |
| 8%|██████████████████▍ | 4680/60000 [13:58:16<142:21:32, 9.26s/it] | |
| 8%|██████████████████▍ | 4700/60000 [14:01:20<139:31:05, 9.08s/it] | |
| 8%|██████████████████▌ | 4720/60000 [14:04:23<140:58:48, 9.18s/it] | |
| 8%|██████████████████▋ | 4740/60000 [14:07:47<142:14:29, 9.27s/it] | |
| 8%|██████████████████▋ | 4760/60000 [14:10:50<141:40:34, 9.23s/it] | |
| 8%|██████████████████▊ | 4780/60000 [14:13:53<139:52:31, 9.12s/it] | |
| 8%|██████████████████▉ | 4800/60000 [14:16:55<138:27:32, 9.03s/it] | |
| 8%|██████████████████▉ | 4820/60000 [14:19:59<141:38:49, 9.24s/it] | |
| 8%|███████████████████ | 4840/60000 [14:23:13<154:09:08, 10.06s/it] | |
| 8%|███████████████████ | 4859/60000 [14:26:14<141:10:51, 9.22s/it] | |
| 8%|███████████████████▏ | 4880/60000 [14:29:29<141:33:20, 9.25s/it] | |
| 8%|███████████████████▎ | 4899/60000 [14:32:25<138:47:05, 9.07s/it] | |
| 8%|███████████████████▎ | 4919/60000 [14:35:34<142:20:01, 9.30s/it] | |
| Reading metadata...: 1it [00:00, 2.55it/s] | |
| 8%|███████████████████▍ | 4939/60000 [14:38:35<137:08:39, 8.97s/it] | |
| 8%|███████████████████▌ | 4959/60000 [14:41:37<138:07:06, 9.03s/it] | |
| 8%|███████████████████▌ | 4979/60000 [14:44:51<136:58:29, 8.96s/it] | |
| 8%|███████████████████▋ | 4999/60000 [14:47:58<161:55:31, 10.60s/it] | |
| 8%|███████████████████▋ | 5000/60000 [14:48:07<154:23:38, 10.11s/it][INFO|trainer.py:3173] 2023-11-19 04:08:31,335 >> ***** Running Evaluation ***** | |
| [INFO|trainer.py:3177] 2023-11-19 04:08:31,336 >> Num examples: Unknown | |
| [INFO|trainer.py:3178] 2023-11-19 04:08:31,336 >> Batch size = 4 | |
| Reading metadata...: 1704it [00:00, 10489.07it/s] | |
| [INFO|trainer_utils.py:759] 2023-11-19 04:08:32,633 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale. If input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| {'eval_loss': 0.13427734375, 'eval_wer': 8.876743309460988, 'eval_runtime': 590.5226, 'eval_samples_per_second': 2.886, 'eval_steps_per_second': 0.721, 'epoch': 0.08} | |
| 8%|███████████████████▋ | 5000/60000 [14:57:58<154:23:38, 10.11s/it][INFO|trainer.py:2896] 2023-11-19 04:18:48,757 >> Saving model checkpoint to ./checkpoint-5000 | |
| [INFO|configuration_utils.py:462] 2023-11-19 04:18:48,768 >> Configuration saved in ./checkpoint-5000/config.json | |
| [INFO|configuration_utils.py:568] 2023-11-19 04:18:48,773 >> Configuration saved in ./checkpoint-5000/generation_config.json | |
| [2023-11-19 04:19:36,452] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step5000 is about to be saved! | |
| [2023-11-19 04:19:36,477] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-5000/global_step5000/mp_rank_00_model_states.pt | |
| [2023-11-19 04:19:36,477] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-5000/global_step5000/mp_rank_00_model_states.pt... | |
| [INFO|modeling_utils.py:2194] 2023-11-19 04:19:36,434 >> Model weights saved in ./checkpoint-5000/pytorch_model.bin | |
| [INFO|feature_extraction_utils.py:425] 2023-11-19 04:19:36,438 >> Feature extractor saved in ./checkpoint-5000/preprocessor_config.json | |
| [2023-11-19 04:19:45,104] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-5000/global_step5000/mp_rank_00_model_states.pt. | |
| [2023-11-19 04:19:45,108] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-5000/global_step5000/zero_pp_rank_0_mp_rank_00_optim_states.pt... | |
| [2023-11-19 04:20:11,141] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-5000/global_step5000/zero_pp_rank_0_mp_rank_00_optim_states.pt. | |
| [2023-11-19 04:20:11,150] [INFO] [engine.py:3417:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-5000/global_step5000/zero_pp_rank_0_mp_rank_00_optim_states.pt | |
| [2023-11-19 04:20:11,150] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step5000 is ready now! | |
| [INFO|feature_extraction_utils.py:425] 2023-11-19 04:21:08,374 >> Feature extractor saved in ./preprocessor_config.json | |
| 8%|███████████████████▋ | 5020/60000 [15:03:55<151:26:59, 9.92s/it] | |
| 8%|███████████████████▊ | 5039/60000 [15:06:53<141:57:34, 9.30s/it] | |
| 8%|███████████████████▉ | 5060/60000 [15:10:10<144:48:47, 9.49s/it] | |
| 8%|███████████████████▉ | 5079/60000 [15:13:10<142:38:34, 9.35s/it] | |
| 8%|████████████████████ | 5099/60000 [15:16:17<138:15:38, 9.07s/it] | |
| 9%|████████████████████▏ | 5119/60000 [15:19:49<186:27:32, 12.23s/it] | |
| 9%|████████████████████▏ | 5140/60000 [15:23:05<140:57:48, 9.25s/it] | |
| 9%|████████████████████▎ | 5159/60000 [15:26:01<141:07:57, 9.26s/it] | |
| Reading metadata...: 2165it [00:00, 13143.40it/s] | 5165/60000 [15:26:56<140:01:37, 9.19s/it] | |
| 9%|████████████████████▎ | 5180/60000 [15:29:22<147:19:27, 9.67s/it] | |
| 9%|████████████████████▍ | 5184/60000 [15:29:55<126:22:47, 8.30s/it] | |
| 9%|████████████████████▍ | 5199/60000 [15:32:13<141:17:06, 9.28s/it] | |
| 9%|████████████████████▌ | 5219/60000 [15:35:18<138:39:00, 9.11s/it] | |
| 9%|████████████████████▌ | 5240/60000 [15:38:44<142:33:38, 9.37s/it] | |
| 9%|████████████████████▋ | 5259/60000 [15:41:40<138:42:07, 9.12s/it] | |
| 9%|████████████████████▊ | 5279/60000 [15:44:50<139:40:03, 9.19s/it] | |
| 9%|████████████████████▊ | 5299/60000 [15:47:54<143:46:47, 9.46s/it] | |
| 9%|████████████████████▉ | 5319/60000 [15:51:13<140:19:23, 9.24s/it] | |
| 9%|█████████████████████ | 5340/60000 [15:55:01<143:07:48, 9.43s/it] | |
| 9%|█████████████████████ | 5360/60000 [15:58:12<142:34:52, 9.39s/it] | |
| 9%|█████████████████████▏ | 5380/60000 [16:01:16<138:08:35, 9.11s/it] | |
| 9%|█████████████████████▏ | 5400/60000 [16:04:22<144:24:15, 9.52s/it] | |
| 9%|█████████████████████▎ | 5420/60000 [16:07:50<157:42:27, 10.40s/it] | |
| 9%|█████████████████████▍ | 5440/60000 [16:11:02<156:41:23, 10.34s/it] | |
| 9%|█████████████████████▍ | 5460/60000 [16:14:06<140:40:02, 9.28s/it] | |
| 9%|█████████████████████▌ | 5480/60000 [16:17:07<135:30:01, 8.95s/it] | |
| 9%|█████████████████████▋ | 5500/60000 [16:20:12<139:29:05, 9.21s/it] | |
| 9%|█████████████████████▋ | 5520/60000 [16:23:17<139:41:46, 9.23s/it] | |
| 9%|█████████████████████▊ | 5539/60000 [16:26:15<138:57:22, 9.19s/it] | |
| 9%|█████████████████████▊ | 5560/60000 [16:29:26<136:37:37, 9.03s/it] | |
| 9%|█████████████████████▉ | 5580/60000 [16:32:26<135:34:52, 8.97s/it] | |
| 9%|██████████████████████ | 5600/60000 [16:35:27<136:14:29, 9.02s/it] | |
| 9%|██████████████████████ | 5619/60000 [16:38:24<144:47:43, 9.59s/it] | |
| 9%|██████████████████████▏ | 5639/60000 [16:41:44<142:26:22, 9.43s/it] | |
| 9%|██████████████████████▎ | 5660/60000 [16:44:55<136:47:25, 9.06s/it] | |
| 9%|██████████████████████▎ | 5680/60000 [16:48:15<161:38:48, 10.71s/it] | |
| 9%|██████████████████████▍ | 5699/60000 [16:51:10<139:03:35, 9.22s/it] | |
| 10%|██████████████████████▍ | 5720/60000 [16:54:27<135:50:07, 9.01s/it] | |
| 10%|██████████████████████▌ | 5740/60000 [16:57:26<136:37:25, 9.06s/it] | |
| 10%|██████████████████████▋ | 5760/60000 [17:00:30<139:16:32, 9.24s/it] | |
| 10%|██████████████████████▋ | 5780/60000 [17:03:34<138:17:50, 9.18s/it] | |
| 10%|██████████████████████▊ | 5800/60000 [17:06:42<138:04:03, 9.17s/it] | |
| 10%|██████████████████████▉ | 5820/60000 [17:09:43<135:29:11, 9.00s/it] | |
| 10%|██████████████████████▉ | 5839/60000 [17:12:37<136:53:55, 9.10s/it] | |
| 10%|███████████████████████ | 5859/60000 [17:16:20<154:49:23, 10.29s/it] | |
| 10%|███████████████████████ | 5879/60000 [17:19:28<145:28:07, 9.68s/it] | |
| 10%|███████████████████████▏ | 5899/60000 [17:22:36<137:05:28, 9.12s/it] | |
| Reading metadata...: 1650it [00:00, 10583.89it/s] | 5910/60000 [17:24:15<135:54:39, 9.05s/it] | |
| 10%|███████████████████████▎ | 5919/60000 [17:25:40<138:56:39, 9.25s/it] | |
| 10%|███████████████████████▎ | 5939/60000 [17:28:43<138:23:48, 9.22s/it] | |
| 10%|███████████████████████▍ | 5959/60000 [17:31:44<134:51:49, 8.98s/it] | |
| 10%|███████████████████████▌ | 5979/60000 [17:35:03<136:08:23, 9.07s/it] | |
| 10%|███████████████████████▌ | 6000/60000 [17:38:28<137:45:52, 9.18s/it][INFO|trainer.py:3173] 2023-11-19 06:58:51,907 >> ***** Running Evaluation ***** | |
| [INFO|trainer.py:3177] 2023-11-19 06:58:51,911 >> Num examples: Unknown | |
| [INFO|trainer.py:3178] 2023-11-19 06:58:51,911 >> Batch size = 4 | |
| Reading metadata...: 1704it [00:00, 9946.01it/s] | |
| Reading metadata...: 1it [00:00, 6.69it/s] | |
| [INFO|trainer_utils.py:759] 2023-11-19 06:58:52,895 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale. If input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| {'eval_loss': 0.1368408203125, 'eval_wer': 8.876743309460988, 'eval_runtime': 591.8673, 'eval_samples_per_second': 2.879, 'eval_steps_per_second': 0.72, 'epoch': 0.1} | |
| 10%|███████████████████████▌ | 6000/60000 [17:48:20<137:45:52, 9.18s/it][INFO|trainer.py:2896] 2023-11-19 07:09:16,613 >> Saving model checkpoint to ./checkpoint-6000 | |
| [INFO|configuration_utils.py:462] 2023-11-19 07:09:16,627 >> Configuration saved in ./checkpoint-6000/config.json | |
| [INFO|configuration_utils.py:568] 2023-11-19 07:09:16,632 >> Configuration saved in ./checkpoint-6000/generation_config.json | |
| [2023-11-19 07:10:00,067] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step6000 is about to be saved! | |
| [2023-11-19 07:10:00,107] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-6000/global_step6000/mp_rank_00_model_states.pt | |
| [2023-11-19 07:10:00,108] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-6000/global_step6000/mp_rank_00_model_states.pt... | |
| [INFO|modeling_utils.py:2194] 2023-11-19 07:10:00,022 >> Model weights saved in ./checkpoint-6000/pytorch_model.bin | |
| [INFO|feature_extraction_utils.py:425] 2023-11-19 07:10:00,031 >> Feature extractor saved in ./checkpoint-6000/preprocessor_config.json | |
| [2023-11-19 07:10:08,388] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-6000/global_step6000/mp_rank_00_model_states.pt. | |
| [2023-11-19 07:10:08,398] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-6000/global_step6000/zero_pp_rank_0_mp_rank_00_optim_states.pt... | |
| [2023-11-19 07:10:29,571] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-6000/global_step6000/zero_pp_rank_0_mp_rank_00_optim_states.pt. | |
| [2023-11-19 07:10:29,584] [INFO] [engine.py:3417:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-6000/global_step6000/zero_pp_rank_0_mp_rank_00_optim_states.pt | |
| [2023-11-19 07:10:29,584] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step6000 is ready now! | |
| [INFO|feature_extraction_utils.py:425] 2023-11-19 07:11:26,721 >> Feature extractor saved in ./preprocessor_config.json | |
| 10%|███████████████████████▋ | 6019/60000 [17:54:07<147:13:43, 9.82s/it] | |
| 10%|███████████████████████▊ | 6039/60000 [17:57:18<140:10:10, 9.35s/it] | |
| 10%|███████████████████████▊ | 6060/60000 [18:00:41<148:15:47, 9.90s/it] | |
| 10%|███████████████████████▉ | 6080/60000 [18:03:49<136:45:44, 9.13s/it] | |
| 10%|███████████████████████▉ | 6099/60000 [18:06:46<138:57:15, 9.28s/it] | |
| 10%|████████████████████████ | 6120/60000 [18:10:02<137:38:52, 9.20s/it] | |
| 10%|████████████████████████▏ | 6140/60000 [18:13:08<141:54:36, 9.49s/it] | |
| 10%|████████████████████████▏ | 6159/60000 [18:16:10<139:33:45, 9.33s/it] | |
| 10%|████████████████████████▎ | 6179/60000 [18:19:14<140:41:45, 9.41s/it] | |
| 10%|████████████████████████▍ | 6199/60000 [18:22:21<142:31:31, 9.54s/it] | |
| 10%|████████████████████████▍ | 6220/60000 [18:25:33<136:35:07, 9.14s/it] | |
| 10%|████████████████████████▌ | 6239/60000 [18:28:35<146:24:14, 9.80s/it] | |
| 10%|████████████████████████▌ | 6260/60000 [18:31:48<135:03:47, 9.05s/it] | |
| 10%|████████████████████████▋ | 6280/60000 [18:34:52<134:42:02, 9.03s/it] | |
| 10%|████████████████████████▊ | 6300/60000 [18:38:14<178:07:41, 11.94s/it] | |
| 11%|████████████████████████▊ | 6319/60000 [18:41:12<136:30:04, 9.15s/it] | |
| 11%|████████████████████████▉ | 6339/60000 [18:44:56<137:07:25, 9.20s/it] | |
| 11%|█████████████████████████ | 6360/60000 [18:48:09<137:16:45, 9.21s/it] | |
| 11%|█████████████████████████ | 6380/60000 [18:51:13<136:51:39, 9.19s/it] | |
| 11%|█████████████████████████▏ | 6400/60000 [18:54:18<137:26:33, 9.23s/it] | |
| 11%|█████████████████████████▎ | 6420/60000 [18:57:26<137:32:23, 9.24s/it] | |
| 11%|█████████████████████████▎ | 6440/60000 [19:00:28<135:19:39, 9.10s/it] | |
| 11%|█████████████████████████▍ | 6460/60000 [19:03:35<153:07:22, 10.30s/it] | |
| Reading metadata...: 2165it [00:00, 12982.32it/s] | 6464/60000 [19:04:18<155:13:19, 10.44s/it] | |
| 11%|█████████████████████████▍ | 6480/60000 [19:06:54<143:38:53, 9.66s/it] | |
| 11%|█████████████████████████▌ | 6500/60000 [19:10:07<148:01:49, 9.96s/it] | |
| 11%|█████████████████████████▋ | 6520/60000 [19:13:16<142:11:29, 9.57s/it] | |
| 11%|█████████████████████████▋ | 6540/60000 [19:16:23<143:35:52, 9.67s/it] | |
| 11%|█████████████████████████▊ | 6560/60000 [19:19:28<136:50:54, 9.22s/it] | |
| 11%|█████████████████████████▉ | 6580/60000 [19:22:33<138:23:59, 9.33s/it] | |
| 11%|█████████████████████████▉ | 6600/60000 [19:26:09<137:35:52, 9.28s/it] | |
| 11%|██████████████████████████ | 6620/60000 [19:29:13<135:27:06, 9.13s/it] | |
| 11%|██████████████████████████ | 6640/60000 [19:33:06<143:51:38, 9.71s/it] | |
| 11%|██████████████████████████▏ | 6659/60000 [19:36:01<136:15:03, 9.20s/it] | |
| 11%|██████████████████████████▎ | 6679/60000 [19:39:12<144:17:09, 9.74s/it] | |
| 11%|██████████████████████████▎ | 6699/60000 [19:42:18<133:33:33, 9.02s/it] | |
| 11%|██████████████████████████▍ | 6719/60000 [19:45:22<137:27:06, 9.29s/it] | |
| 11%|██████████████████████████▌ | 6740/60000 [19:48:49<143:07:45, 9.67s/it] | |
| 11%|██████████████████████████▌ | 6760/60000 [19:51:53<137:12:21, 9.28s/it] | |
| 11%|██████████████████████████▋ | 6779/60000 [19:54:52<135:29:24, 9.16s/it] | |
| 11%|██████████████████████████▋ | 6799/60000 [19:57:56<135:05:53, 9.14s/it] | |
| 11%|██████████████████████████▊ | 6819/60000 [20:00:59<137:10:24, 9.29s/it] | |
| 11%|██████████████████████████▉ | 6839/60000 [20:04:03<133:52:03, 9.07s/it] | |
| 11%|██████████████████████████▉ | 6860/60000 [20:07:22<134:15:04, 9.09s/it] | |
| 11%|███████████████████████████ | 6880/60000 [20:10:28<141:11:17, 9.57s/it] | |
| Reading metadata...: 1650it [00:00, 4019.95it/s] | 6899/60000 [20:13:26<135:28:47, 9.18s/it] | |
| 12%|███████████████████████████▏ | 6900/60000 [20:13:37<143:28:17, 9.73s/it] | |
| 12%|███████████████████████████▏ | 6919/60000 [20:16:36<138:08:56, 9.37s/it] | |
| 12%|███████████████████████████▎ | 6940/60000 [20:19:57<155:15:25, 10.53s/it] | |
| 12%|███████████████████████████▎ | 6959/60000 [20:22:53<136:55:04, 9.29s/it] | |
| 12%|███████████████████████████▍ | 6979/60000 [20:26:13<139:06:06, 9.44s/it] | |
| 12%|███████████████████████████▌ | 7000/60000 [20:29:27<135:59:18, 9.24s/it][INFO|trainer.py:3173] 2023-11-19 09:49:51,427 >> ***** Running Evaluation ***** | |
| [INFO|trainer.py:3177] 2023-11-19 09:49:51,427 >> Num examples: Unknown | |
| [INFO|trainer.py:3178] 2023-11-19 09:49:51,428 >> Batch size = 4 | |
| Reading metadata...: 1704it [00:00, 10117.74it/s] | |
| Reading metadata...: 1it [00:00, 6.90it/s] | |
| [INFO|trainer_utils.py:759] 2023-11-19 09:49:52,356 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale. If input_length, segment, accent, up_votes, age, path, client_id, down_votes, gender, locale are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| 12%|███████████████████████████▌ | 7000/60000 [20:40:36<135:59:18, 9.24s/it] | |
| 12%|███████████████████████████▌ | 7000/60000 [20:40:36<135:59:18, 9.24s/it][INFO|trainer.py:2896] 2023-11-19 10:01:42,953 >> Saving model checkpoint to ./checkpoint-7000 | |
| [INFO|configuration_utils.py:462] 2023-11-19 10:01:42,964 >> Configuration saved in ./checkpoint-7000/config.json | |
| [INFO|configuration_utils.py:568] 2023-11-19 10:01:42,971 >> Configuration saved in ./checkpoint-7000/generation_config.json | |
| [2023-11-19 10:02:35,205] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step7000 is about to be saved! | |
| [2023-11-19 10:02:35,238] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-7000/global_step7000/mp_rank_00_model_states.pt | |
| [2023-11-19 10:02:35,238] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-7000/global_step7000/mp_rank_00_model_states.pt... | |
| [INFO|modeling_utils.py:2194] 2023-11-19 10:02:35,175 >> Model weights saved in ./checkpoint-7000/pytorch_model.bin | |
| [INFO|feature_extraction_utils.py:425] 2023-11-19 10:02:35,181 >> Feature extractor saved in ./checkpoint-7000/preprocessor_config.json | |
| [2023-11-19 10:02:56,297] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-7000/global_step7000/mp_rank_00_model_states.pt. | |
| [2023-11-19 10:02:56,319] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-7000/global_step7000/zero_pp_rank_0_mp_rank_00_optim_states.pt... | |
| [2023-11-19 10:04:10,359] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-7000/global_step7000/zero_pp_rank_0_mp_rank_00_optim_states.pt. | |
| [2023-11-19 10:04:10,369] [INFO] [engine.py:3417:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-7000/global_step7000/zero_pp_rank_0_mp_rank_00_optim_states.pt | |
| [2023-11-19 10:04:10,370] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step7000 is ready now! | |
| [INFO|feature_extraction_utils.py:425] 2023-11-19 10:05:15,419 >> Feature extractor saved in ./preprocessor_config.json | |
| 12%|███████████████████████████▌ | 7020/60000 [20:48:31<159:37:27, 10.85s/it] | |
| 12%|███████████████████████████▋ | 7040/60000 [20:52:05<149:28:41, 10.16s/it] | |
| 12%|███████████████████████████▊ | 7059/60000 [20:55:20<151:14:07, 10.28s/it] | |
| 12%|███████████████████████████▊ | 7079/60000 [20:58:45<159:22:43, 10.84s/it] | |