|
2022-02-01 15:08:19,400 INFO MainThread:3248864 [internal.py:wandb_internal():87] W&B internal server running at pid: 3248864, started at: 2022-02-01 15:08:19.399884 |
|
2022-02-01 15:08:19,402 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: check_version |
|
2022-02-01 15:08:19,402 INFO WriterThread:3248864 [datastore.py:open_for_write():77] open: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb |
|
2022-02-01 15:08:19,405 DEBUG SenderThread:3248864 [sender.py:send():234] send: header |
|
2022-02-01 15:08:19,405 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: check_version |
|
2022-02-01 15:08:19,562 DEBUG SenderThread:3248864 [sender.py:send():234] send: run |
|
2022-02-01 15:08:19,847 INFO SenderThread:3248864 [dir_watcher.py:__init__():169] watching files in: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files |
|
2022-02-01 15:08:19,848 INFO SenderThread:3248864 [sender.py:_start_run_threads():804] run started: 33gv5m8t with start time 1643724498 |
|
2022-02-01 15:08:19,848 DEBUG SenderThread:3248864 [sender.py:send():234] send: summary |
|
2022-02-01 15:08:19,848 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end |
|
2022-02-01 15:08:19,850 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: run_start |
|
2022-02-01 15:08:20,854 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json |
|
2022-02-01 15:08:21,089 DEBUG HandlerThread:3248864 [meta.py:__init__():40] meta init |
|
2022-02-01 15:08:21,090 DEBUG HandlerThread:3248864 [meta.py:__init__():54] meta init done |
|
2022-02-01 15:08:21,090 DEBUG HandlerThread:3248864 [meta.py:probe():214] probe |
|
2022-02-01 15:08:21,096 DEBUG HandlerThread:3248864 [meta.py:_setup_git():204] setup git |
|
2022-02-01 15:08:21,129 DEBUG HandlerThread:3248864 [meta.py:_setup_git():211] setup git done |
|
2022-02-01 15:08:21,130 DEBUG HandlerThread:3248864 [meta.py:_save_code():92] save code |
|
2022-02-01 15:08:21,144 DEBUG HandlerThread:3248864 [meta.py:_save_code():113] save code done |
|
2022-02-01 15:08:21,144 DEBUG HandlerThread:3248864 [meta.py:_save_patches():130] save patches |
|
2022-02-01 15:08:21,223 DEBUG HandlerThread:3248864 [meta.py:_save_patches():172] save patches done |
|
2022-02-01 15:08:21,223 DEBUG HandlerThread:3248864 [meta.py:_save_pip():58] save pip |
|
2022-02-01 15:08:21,224 DEBUG HandlerThread:3248864 [meta.py:_save_pip():72] save pip done |
|
2022-02-01 15:08:21,224 DEBUG HandlerThread:3248864 [meta.py:probe():252] probe done |
|
2022-02-01 15:08:21,230 DEBUG SenderThread:3248864 [sender.py:send():234] send: files |
|
2022-02-01 15:08:21,231 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now |
|
2022-02-01 15:08:21,232 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file code/run_speech_recognition_ctc.py with policy now |
|
2022-02-01 15:08:21,240 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-02-01 15:08:21,245 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: stop_status |
|
2022-02-01 15:08:21,493 DEBUG SenderThread:3248864 [sender.py:send():234] send: config |
|
2022-02-01 15:08:21,495 DEBUG SenderThread:3248864 [sender.py:send():234] send: metric |
|
2022-02-01 15:08:21,496 DEBUG SenderThread:3248864 [sender.py:send():234] send: metric |
|
2022-02-01 15:08:21,496 WARNING SenderThread:3248864 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen) |
|
2022-02-01 15:08:21,852 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json |
|
2022-02-01 15:08:21,852 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py |
|
2022-02-01 15:08:21,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt |
|
2022-02-01 15:08:21,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log |
|
2022-02-01 15:08:21,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code |
|
2022-02-01 15:08:21,960 INFO Thread-11 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/bkne7cxj-wandb-metadata.json |
|
2022-02-01 15:08:21,969 INFO Thread-12 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/1l3kkiz5-code/run_speech_recognition_ctc.py |
|
2022-02-01 15:08:23,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log |
|
2022-02-01 15:08:25,854 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log |
|
2022-02-01 15:08:25,932 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:25,933 DEBUG SenderThread:3248864 [sender.py:send():234] send: telemetry |
|
2022-02-01 15:08:25,933 DEBUG SenderThread:3248864 [sender.py:send():234] send: exit |
|
2022-02-01 15:08:25,933 INFO SenderThread:3248864 [sender.py:send_exit():366] handling exit code: 1 |
|
2022-02-01 15:08:25,934 INFO SenderThread:3248864 [sender.py:send_exit():368] handling runtime: 6 |
|
2022-02-01 15:08:25,934 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end |
|
2022-02-01 15:08:25,934 INFO SenderThread:3248864 [sender.py:send_exit():374] send defer |
|
2022-02-01 15:08:25,935 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:25,936 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:25,936 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 0 |
|
2022-02-01 15:08:25,936 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:25,936 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 0 |
|
2022-02-01 15:08:25,936 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 1 |
|
2022-02-01 15:08:25,937 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:25,937 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 1 |
|
2022-02-01 15:08:26,020 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:26,020 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 1 |
|
2022-02-01 15:08:26,020 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 2 |
|
2022-02-01 15:08:26,021 DEBUG SenderThread:3248864 [sender.py:send():234] send: stats |
|
2022-02-01 15:08:26,022 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:26,022 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 2 |
|
2022-02-01 15:08:26,022 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:26,022 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 2 |
|
2022-02-01 15:08:26,022 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 3 |
|
2022-02-01 15:08:26,023 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:26,023 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 3 |
|
2022-02-01 15:08:26,023 DEBUG SenderThread:3248864 [sender.py:send():234] send: summary |
|
2022-02-01 15:08:26,024 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end |
|
2022-02-01 15:08:26,024 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:26,024 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 3 |
|
2022-02-01 15:08:26,024 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 4 |
|
2022-02-01 15:08:26,025 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:26,025 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 4 |
|
2022-02-01 15:08:26,025 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:26,026 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 4 |
|
2022-02-01 15:08:26,038 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:26,367 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 5 |
|
2022-02-01 15:08:26,367 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:26,368 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:26,368 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 5 |
|
2022-02-01 15:08:26,368 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:26,368 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 5 |
|
2022-02-01 15:08:26,368 INFO SenderThread:3248864 [dir_watcher.py:finish():283] shutting down directory watcher |
|
2022-02-01 15:08:26,469 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:26,854 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json |
|
2022-02-01 15:08:26,855 INFO SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml |
|
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log |
|
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:finish():313] scan: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files |
|
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml config.yaml |
|
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt requirements.txt |
|
2022-02-01 15:08:26,857 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json wandb-summary.json |
|
2022-02-01 15:08:26,857 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log output.log |
|
2022-02-01 15:08:26,857 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json wandb-metadata.json |
|
2022-02-01 15:08:26,869 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py code/run_speech_recognition_ctc.py |
|
2022-02-01 15:08:26,869 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 6 |
|
2022-02-01 15:08:26,869 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:26,881 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:26,881 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 6 |
|
2022-02-01 15:08:26,882 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:26,882 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 6 |
|
2022-02-01 15:08:26,882 INFO SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher |
|
2022-02-01 15:08:26,984 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:26,984 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:27,086 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:27,087 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:27,189 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:27,189 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:27,291 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:27,292 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:27,394 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:27,394 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:27,497 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:27,497 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:27,545 INFO Thread-13 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml |
|
2022-02-01 15:08:27,551 INFO Thread-14 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt |
|
2022-02-01 15:08:27,553 INFO Thread-16 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log |
|
2022-02-01 15:08:27,565 INFO Thread-15 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json |
|
2022-02-01 15:08:27,599 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:27,600 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:27,702 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:27,702 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:27,766 INFO Thread-7 :3248864 [sender.py:transition_state():387] send defer: 7 |
|
2022-02-01 15:08:27,767 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:27,767 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 7 |
|
2022-02-01 15:08:27,767 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:27,767 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 7 |
|
2022-02-01 15:08:27,805 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:28,272 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 8 |
|
2022-02-01 15:08:28,272 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:28,273 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:28,273 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 8 |
|
2022-02-01 15:08:28,274 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:28,274 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 8 |
|
2022-02-01 15:08:28,274 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 9 |
|
2022-02-01 15:08:28,275 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer |
|
2022-02-01 15:08:28,275 DEBUG SenderThread:3248864 [sender.py:send():234] send: final |
|
2022-02-01 15:08:28,275 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 9 |
|
2022-02-01 15:08:28,276 DEBUG SenderThread:3248864 [sender.py:send():234] send: footer |
|
2022-02-01 15:08:28,276 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer |
|
2022-02-01 15:08:28,276 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 9 |
|
2022-02-01 15:08:28,375 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-02-01 15:08:28,375 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit |
|
2022-02-01 15:08:28,375 INFO SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher |
|
2022-02-01 15:08:28,858 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: get_summary |
|
2022-02-01 15:08:28,859 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: sampled_history |
|
2022-02-01 15:08:28,861 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: shutdown |
|
2022-02-01 15:08:28,861 INFO HandlerThread:3248864 [handler.py:finish():731] shutting down handler |
|
2022-02-01 15:08:29,275 INFO WriterThread:3248864 [datastore.py:close():281] close: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb |
|
2022-02-01 15:08:29,856 INFO SenderThread:3248864 [sender.py:finish():1070] shutting down sender |
|
2022-02-01 15:08:29,856 INFO SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher |
|
2022-02-01 15:08:29,856 INFO SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher |
|
2022-02-01 15:08:29,859 INFO MainThread:3248864 [internal.py:handle_exit():77] Internal process exited |
|
|