versae's picture
Training in progress, step 500
981dcc0
raw
history blame contribute delete
No virus
19.2 kB
2022-02-01 15:08:19,400 INFO MainThread:3248864 [internal.py:wandb_internal():87] W&B internal server running at pid: 3248864, started at: 2022-02-01 15:08:19.399884
2022-02-01 15:08:19,402 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 15:08:19,402 INFO WriterThread:3248864 [datastore.py:open_for_write():77] open: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb
2022-02-01 15:08:19,405 DEBUG SenderThread:3248864 [sender.py:send():234] send: header
2022-02-01 15:08:19,405 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: check_version
2022-02-01 15:08:19,562 DEBUG SenderThread:3248864 [sender.py:send():234] send: run
2022-02-01 15:08:19,847 INFO SenderThread:3248864 [dir_watcher.py:__init__():169] watching files in: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files
2022-02-01 15:08:19,848 INFO SenderThread:3248864 [sender.py:_start_run_threads():804] run started: 33gv5m8t with start time 1643724498
2022-02-01 15:08:19,848 DEBUG SenderThread:3248864 [sender.py:send():234] send: summary
2022-02-01 15:08:19,848 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:19,850 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 15:08:20,854 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:21,089 DEBUG HandlerThread:3248864 [meta.py:__init__():40] meta init
2022-02-01 15:08:21,090 DEBUG HandlerThread:3248864 [meta.py:__init__():54] meta init done
2022-02-01 15:08:21,090 DEBUG HandlerThread:3248864 [meta.py:probe():214] probe
2022-02-01 15:08:21,096 DEBUG HandlerThread:3248864 [meta.py:_setup_git():204] setup git
2022-02-01 15:08:21,129 DEBUG HandlerThread:3248864 [meta.py:_setup_git():211] setup git done
2022-02-01 15:08:21,130 DEBUG HandlerThread:3248864 [meta.py:_save_code():92] save code
2022-02-01 15:08:21,144 DEBUG HandlerThread:3248864 [meta.py:_save_code():113] save code done
2022-02-01 15:08:21,144 DEBUG HandlerThread:3248864 [meta.py:_save_patches():130] save patches
2022-02-01 15:08:21,223 DEBUG HandlerThread:3248864 [meta.py:_save_patches():172] save patches done
2022-02-01 15:08:21,223 DEBUG HandlerThread:3248864 [meta.py:_save_pip():58] save pip
2022-02-01 15:08:21,224 DEBUG HandlerThread:3248864 [meta.py:_save_pip():72] save pip done
2022-02-01 15:08:21,224 DEBUG HandlerThread:3248864 [meta.py:probe():252] probe done
2022-02-01 15:08:21,230 DEBUG SenderThread:3248864 [sender.py:send():234] send: files
2022-02-01 15:08:21,231 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 15:08:21,232 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file code/run_speech_recognition_ctc.py with policy now
2022-02-01 15:08:21,240 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 15:08:21,245 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: stop_status
2022-02-01 15:08:21,493 DEBUG SenderThread:3248864 [sender.py:send():234] send: config
2022-02-01 15:08:21,495 DEBUG SenderThread:3248864 [sender.py:send():234] send: metric
2022-02-01 15:08:21,496 DEBUG SenderThread:3248864 [sender.py:send():234] send: metric
2022-02-01 15:08:21,496 WARNING SenderThread:3248864 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 15:08:21,852 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json
2022-02-01 15:08:21,852 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py
2022-02-01 15:08:21,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt
2022-02-01 15:08:21,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:21,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code
2022-02-01 15:08:21,960 INFO Thread-11 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/bkne7cxj-wandb-metadata.json
2022-02-01 15:08:21,969 INFO Thread-12 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/1l3kkiz5-code/run_speech_recognition_ctc.py
2022-02-01 15:08:23,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:25,854 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:25,932 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:25,933 DEBUG SenderThread:3248864 [sender.py:send():234] send: telemetry
2022-02-01 15:08:25,933 DEBUG SenderThread:3248864 [sender.py:send():234] send: exit
2022-02-01 15:08:25,933 INFO SenderThread:3248864 [sender.py:send_exit():366] handling exit code: 1
2022-02-01 15:08:25,934 INFO SenderThread:3248864 [sender.py:send_exit():368] handling runtime: 6
2022-02-01 15:08:25,934 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:25,934 INFO SenderThread:3248864 [sender.py:send_exit():374] send defer
2022-02-01 15:08:25,935 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:25,936 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:25,936 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 0
2022-02-01 15:08:25,936 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:25,936 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 0
2022-02-01 15:08:25,936 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 1
2022-02-01 15:08:25,937 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:25,937 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 1
2022-02-01 15:08:26,020 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,020 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 1
2022-02-01 15:08:26,020 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 2
2022-02-01 15:08:26,021 DEBUG SenderThread:3248864 [sender.py:send():234] send: stats
2022-02-01 15:08:26,022 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,022 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 2
2022-02-01 15:08:26,022 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,022 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 2
2022-02-01 15:08:26,022 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 3
2022-02-01 15:08:26,023 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,023 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 3
2022-02-01 15:08:26,023 DEBUG SenderThread:3248864 [sender.py:send():234] send: summary
2022-02-01 15:08:26,024 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:26,024 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,024 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 3
2022-02-01 15:08:26,024 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 4
2022-02-01 15:08:26,025 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,025 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 4
2022-02-01 15:08:26,025 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,026 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 4
2022-02-01 15:08:26,038 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,367 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 5
2022-02-01 15:08:26,367 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:26,368 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,368 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 5
2022-02-01 15:08:26,368 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,368 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 5
2022-02-01 15:08:26,368 INFO SenderThread:3248864 [dir_watcher.py:finish():283] shutting down directory watcher
2022-02-01 15:08:26,469 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,854 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:26,855 INFO SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:finish():313] scan: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml config.yaml
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt requirements.txt
2022-02-01 15:08:26,857 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json wandb-summary.json
2022-02-01 15:08:26,857 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log output.log
2022-02-01 15:08:26,857 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json wandb-metadata.json
2022-02-01 15:08:26,869 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py code/run_speech_recognition_ctc.py
2022-02-01 15:08:26,869 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 6
2022-02-01 15:08:26,869 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:26,881 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,881 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 6
2022-02-01 15:08:26,882 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,882 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 6
2022-02-01 15:08:26,882 INFO SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 15:08:26,984 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,984 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,086 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,087 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,189 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,189 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,291 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,292 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,394 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,394 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,497 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,497 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,545 INFO Thread-13 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml
2022-02-01 15:08:27,551 INFO Thread-14 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt
2022-02-01 15:08:27,553 INFO Thread-16 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:27,565 INFO Thread-15 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:27,599 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,600 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,702 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,702 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,766 INFO Thread-7 :3248864 [sender.py:transition_state():387] send defer: 7
2022-02-01 15:08:27,767 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:27,767 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 7
2022-02-01 15:08:27,767 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:27,767 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 7
2022-02-01 15:08:27,805 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:28,272 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 8
2022-02-01 15:08:28,272 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:28,273 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:28,273 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 8
2022-02-01 15:08:28,274 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:28,274 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 8
2022-02-01 15:08:28,274 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 9
2022-02-01 15:08:28,275 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:28,275 DEBUG SenderThread:3248864 [sender.py:send():234] send: final
2022-02-01 15:08:28,275 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 9
2022-02-01 15:08:28,276 DEBUG SenderThread:3248864 [sender.py:send():234] send: footer
2022-02-01 15:08:28,276 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:28,276 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 9
2022-02-01 15:08:28,375 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:28,375 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:28,375 INFO SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 15:08:28,858 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: get_summary
2022-02-01 15:08:28,859 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: sampled_history
2022-02-01 15:08:28,861 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: shutdown
2022-02-01 15:08:28,861 INFO HandlerThread:3248864 [handler.py:finish():731] shutting down handler
2022-02-01 15:08:29,275 INFO WriterThread:3248864 [datastore.py:close():281] close: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb
2022-02-01 15:08:29,856 INFO SenderThread:3248864 [sender.py:finish():1070] shutting down sender
2022-02-01 15:08:29,856 INFO SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 15:08:29,856 INFO SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 15:08:29,859 INFO MainThread:3248864 [internal.py:handle_exit():77] Internal process exited