2022-02-01 15:08:19,400 INFO    MainThread:3248864 [internal.py:wandb_internal():87] W&B internal server running at pid: 3248864, started at: 2022-02-01 15:08:19.399884
2022-02-01 15:08:19,402 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 15:08:19,402 INFO    WriterThread:3248864 [datastore.py:open_for_write():77] open: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb
2022-02-01 15:08:19,405 DEBUG   SenderThread:3248864 [sender.py:send():234] send: header
2022-02-01 15:08:19,405 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: check_version
2022-02-01 15:08:19,562 DEBUG   SenderThread:3248864 [sender.py:send():234] send: run
2022-02-01 15:08:19,847 INFO    SenderThread:3248864 [dir_watcher.py:__init__():169] watching files in: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files
2022-02-01 15:08:19,848 INFO    SenderThread:3248864 [sender.py:_start_run_threads():804] run started: 33gv5m8t with start time 1643724498
2022-02-01 15:08:19,848 DEBUG   SenderThread:3248864 [sender.py:send():234] send: summary
2022-02-01 15:08:19,848 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:19,850 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 15:08:20,854 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:21,089 DEBUG   HandlerThread:3248864 [meta.py:__init__():40] meta init
2022-02-01 15:08:21,090 DEBUG   HandlerThread:3248864 [meta.py:__init__():54] meta init done
2022-02-01 15:08:21,090 DEBUG   HandlerThread:3248864 [meta.py:probe():214] probe
2022-02-01 15:08:21,096 DEBUG   HandlerThread:3248864 [meta.py:_setup_git():204] setup git
2022-02-01 15:08:21,129 DEBUG   HandlerThread:3248864 [meta.py:_setup_git():211] setup git done
2022-02-01 15:08:21,130 DEBUG   HandlerThread:3248864 [meta.py:_save_code():92] save code
2022-02-01 15:08:21,144 DEBUG   HandlerThread:3248864 [meta.py:_save_code():113] save code done
2022-02-01 15:08:21,144 DEBUG   HandlerThread:3248864 [meta.py:_save_patches():130] save patches
2022-02-01 15:08:21,223 DEBUG   HandlerThread:3248864 [meta.py:_save_patches():172] save patches done
2022-02-01 15:08:21,223 DEBUG   HandlerThread:3248864 [meta.py:_save_pip():58] save pip
2022-02-01 15:08:21,224 DEBUG   HandlerThread:3248864 [meta.py:_save_pip():72] save pip done
2022-02-01 15:08:21,224 DEBUG   HandlerThread:3248864 [meta.py:probe():252] probe done
2022-02-01 15:08:21,230 DEBUG   SenderThread:3248864 [sender.py:send():234] send: files
2022-02-01 15:08:21,231 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 15:08:21,232 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file code/run_speech_recognition_ctc.py with policy now
2022-02-01 15:08:21,240 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 15:08:21,245 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: stop_status
2022-02-01 15:08:21,493 DEBUG   SenderThread:3248864 [sender.py:send():234] send: config
2022-02-01 15:08:21,495 DEBUG   SenderThread:3248864 [sender.py:send():234] send: metric
2022-02-01 15:08:21,496 DEBUG   SenderThread:3248864 [sender.py:send():234] send: metric
2022-02-01 15:08:21,496 WARNING SenderThread:3248864 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 15:08:21,852 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json
2022-02-01 15:08:21,852 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py
2022-02-01 15:08:21,853 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt
2022-02-01 15:08:21,853 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:21,853 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code
2022-02-01 15:08:21,960 INFO    Thread-11 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/bkne7cxj-wandb-metadata.json
2022-02-01 15:08:21,969 INFO    Thread-12 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/1l3kkiz5-code/run_speech_recognition_ctc.py
2022-02-01 15:08:23,853 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:25,854 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:25,932 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:25,933 DEBUG   SenderThread:3248864 [sender.py:send():234] send: telemetry
2022-02-01 15:08:25,933 DEBUG   SenderThread:3248864 [sender.py:send():234] send: exit
2022-02-01 15:08:25,933 INFO    SenderThread:3248864 [sender.py:send_exit():366] handling exit code: 1
2022-02-01 15:08:25,934 INFO    SenderThread:3248864 [sender.py:send_exit():368] handling runtime: 6
2022-02-01 15:08:25,934 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:25,934 INFO    SenderThread:3248864 [sender.py:send_exit():374] send defer
2022-02-01 15:08:25,935 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:25,936 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:25,936 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 0
2022-02-01 15:08:25,936 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:25,936 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 0
2022-02-01 15:08:25,936 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 1
2022-02-01 15:08:25,937 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:25,937 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 1
2022-02-01 15:08:26,020 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,020 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 1
2022-02-01 15:08:26,020 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 2
2022-02-01 15:08:26,021 DEBUG   SenderThread:3248864 [sender.py:send():234] send: stats
2022-02-01 15:08:26,022 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,022 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 2
2022-02-01 15:08:26,022 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,022 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 2
2022-02-01 15:08:26,022 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 3
2022-02-01 15:08:26,023 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,023 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 3
2022-02-01 15:08:26,023 DEBUG   SenderThread:3248864 [sender.py:send():234] send: summary
2022-02-01 15:08:26,024 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:26,024 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,024 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 3
2022-02-01 15:08:26,024 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 4
2022-02-01 15:08:26,025 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,025 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 4
2022-02-01 15:08:26,025 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,026 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 4
2022-02-01 15:08:26,038 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,367 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 5
2022-02-01 15:08:26,367 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:26,368 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,368 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 5
2022-02-01 15:08:26,368 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,368 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 5
2022-02-01 15:08:26,368 INFO    SenderThread:3248864 [dir_watcher.py:finish():283] shutting down directory watcher
2022-02-01 15:08:26,469 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,854 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:26,855 INFO    SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml
2022-02-01 15:08:26,856 INFO    SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:26,856 INFO    SenderThread:3248864 [dir_watcher.py:finish():313] scan: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files
2022-02-01 15:08:26,856 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml config.yaml
2022-02-01 15:08:26,856 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt requirements.txt
2022-02-01 15:08:26,857 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json wandb-summary.json
2022-02-01 15:08:26,857 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log output.log
2022-02-01 15:08:26,857 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json wandb-metadata.json
2022-02-01 15:08:26,869 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py code/run_speech_recognition_ctc.py
2022-02-01 15:08:26,869 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 6
2022-02-01 15:08:26,869 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:26,881 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,881 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 6
2022-02-01 15:08:26,882 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,882 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 6
2022-02-01 15:08:26,882 INFO    SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 15:08:26,984 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,984 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,086 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,087 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,189 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,189 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,291 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,292 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,394 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,394 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,497 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,497 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,545 INFO    Thread-13 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml
2022-02-01 15:08:27,551 INFO    Thread-14 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt
2022-02-01 15:08:27,553 INFO    Thread-16 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:27,565 INFO    Thread-15 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:27,599 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,600 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,702 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,702 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,766 INFO    Thread-7  :3248864 [sender.py:transition_state():387] send defer: 7
2022-02-01 15:08:27,767 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:27,767 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 7
2022-02-01 15:08:27,767 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:27,767 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 7
2022-02-01 15:08:27,805 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:28,272 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 8
2022-02-01 15:08:28,272 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:28,273 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:28,273 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 8
2022-02-01 15:08:28,274 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:28,274 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 8
2022-02-01 15:08:28,274 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 9
2022-02-01 15:08:28,275 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:28,275 DEBUG   SenderThread:3248864 [sender.py:send():234] send: final
2022-02-01 15:08:28,275 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 9
2022-02-01 15:08:28,276 DEBUG   SenderThread:3248864 [sender.py:send():234] send: footer
2022-02-01 15:08:28,276 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:28,276 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 9
2022-02-01 15:08:28,375 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:28,375 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:28,375 INFO    SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 15:08:28,858 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: get_summary
2022-02-01 15:08:28,859 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: sampled_history
2022-02-01 15:08:28,861 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: shutdown
2022-02-01 15:08:28,861 INFO    HandlerThread:3248864 [handler.py:finish():731] shutting down handler
2022-02-01 15:08:29,275 INFO    WriterThread:3248864 [datastore.py:close():281] close: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb
2022-02-01 15:08:29,856 INFO    SenderThread:3248864 [sender.py:finish():1070] shutting down sender
2022-02-01 15:08:29,856 INFO    SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 15:08:29,856 INFO    SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 15:08:29,859 INFO    MainThread:3248864 [internal.py:handle_exit():77] Internal process exited