File size: 19,167 Bytes
981dcc0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 |
2022-02-01 15:08:19,400 INFO MainThread:3248864 [internal.py:wandb_internal():87] W&B internal server running at pid: 3248864, started at: 2022-02-01 15:08:19.399884
2022-02-01 15:08:19,402 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 15:08:19,402 INFO WriterThread:3248864 [datastore.py:open_for_write():77] open: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb
2022-02-01 15:08:19,405 DEBUG SenderThread:3248864 [sender.py:send():234] send: header
2022-02-01 15:08:19,405 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: check_version
2022-02-01 15:08:19,562 DEBUG SenderThread:3248864 [sender.py:send():234] send: run
2022-02-01 15:08:19,847 INFO SenderThread:3248864 [dir_watcher.py:__init__():169] watching files in: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files
2022-02-01 15:08:19,848 INFO SenderThread:3248864 [sender.py:_start_run_threads():804] run started: 33gv5m8t with start time 1643724498
2022-02-01 15:08:19,848 DEBUG SenderThread:3248864 [sender.py:send():234] send: summary
2022-02-01 15:08:19,848 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:19,850 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 15:08:20,854 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:21,089 DEBUG HandlerThread:3248864 [meta.py:__init__():40] meta init
2022-02-01 15:08:21,090 DEBUG HandlerThread:3248864 [meta.py:__init__():54] meta init done
2022-02-01 15:08:21,090 DEBUG HandlerThread:3248864 [meta.py:probe():214] probe
2022-02-01 15:08:21,096 DEBUG HandlerThread:3248864 [meta.py:_setup_git():204] setup git
2022-02-01 15:08:21,129 DEBUG HandlerThread:3248864 [meta.py:_setup_git():211] setup git done
2022-02-01 15:08:21,130 DEBUG HandlerThread:3248864 [meta.py:_save_code():92] save code
2022-02-01 15:08:21,144 DEBUG HandlerThread:3248864 [meta.py:_save_code():113] save code done
2022-02-01 15:08:21,144 DEBUG HandlerThread:3248864 [meta.py:_save_patches():130] save patches
2022-02-01 15:08:21,223 DEBUG HandlerThread:3248864 [meta.py:_save_patches():172] save patches done
2022-02-01 15:08:21,223 DEBUG HandlerThread:3248864 [meta.py:_save_pip():58] save pip
2022-02-01 15:08:21,224 DEBUG HandlerThread:3248864 [meta.py:_save_pip():72] save pip done
2022-02-01 15:08:21,224 DEBUG HandlerThread:3248864 [meta.py:probe():252] probe done
2022-02-01 15:08:21,230 DEBUG SenderThread:3248864 [sender.py:send():234] send: files
2022-02-01 15:08:21,231 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 15:08:21,232 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file code/run_speech_recognition_ctc.py with policy now
2022-02-01 15:08:21,240 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 15:08:21,245 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: stop_status
2022-02-01 15:08:21,493 DEBUG SenderThread:3248864 [sender.py:send():234] send: config
2022-02-01 15:08:21,495 DEBUG SenderThread:3248864 [sender.py:send():234] send: metric
2022-02-01 15:08:21,496 DEBUG SenderThread:3248864 [sender.py:send():234] send: metric
2022-02-01 15:08:21,496 WARNING SenderThread:3248864 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 15:08:21,852 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json
2022-02-01 15:08:21,852 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py
2022-02-01 15:08:21,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt
2022-02-01 15:08:21,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:21,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code
2022-02-01 15:08:21,960 INFO Thread-11 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/bkne7cxj-wandb-metadata.json
2022-02-01 15:08:21,969 INFO Thread-12 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/1l3kkiz5-code/run_speech_recognition_ctc.py
2022-02-01 15:08:23,853 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:25,854 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:25,932 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:25,933 DEBUG SenderThread:3248864 [sender.py:send():234] send: telemetry
2022-02-01 15:08:25,933 DEBUG SenderThread:3248864 [sender.py:send():234] send: exit
2022-02-01 15:08:25,933 INFO SenderThread:3248864 [sender.py:send_exit():366] handling exit code: 1
2022-02-01 15:08:25,934 INFO SenderThread:3248864 [sender.py:send_exit():368] handling runtime: 6
2022-02-01 15:08:25,934 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:25,934 INFO SenderThread:3248864 [sender.py:send_exit():374] send defer
2022-02-01 15:08:25,935 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:25,936 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:25,936 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 0
2022-02-01 15:08:25,936 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:25,936 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 0
2022-02-01 15:08:25,936 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 1
2022-02-01 15:08:25,937 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:25,937 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 1
2022-02-01 15:08:26,020 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,020 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 1
2022-02-01 15:08:26,020 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 2
2022-02-01 15:08:26,021 DEBUG SenderThread:3248864 [sender.py:send():234] send: stats
2022-02-01 15:08:26,022 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,022 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 2
2022-02-01 15:08:26,022 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,022 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 2
2022-02-01 15:08:26,022 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 3
2022-02-01 15:08:26,023 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,023 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 3
2022-02-01 15:08:26,023 DEBUG SenderThread:3248864 [sender.py:send():234] send: summary
2022-02-01 15:08:26,024 INFO SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:26,024 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,024 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 3
2022-02-01 15:08:26,024 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 4
2022-02-01 15:08:26,025 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,025 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 4
2022-02-01 15:08:26,025 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,026 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 4
2022-02-01 15:08:26,038 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,367 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 5
2022-02-01 15:08:26,367 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:26,368 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,368 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 5
2022-02-01 15:08:26,368 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,368 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 5
2022-02-01 15:08:26,368 INFO SenderThread:3248864 [dir_watcher.py:finish():283] shutting down directory watcher
2022-02-01 15:08:26,469 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,854 INFO Thread-8 :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:26,855 INFO SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:finish():313] scan: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml config.yaml
2022-02-01 15:08:26,856 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt requirements.txt
2022-02-01 15:08:26,857 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json wandb-summary.json
2022-02-01 15:08:26,857 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log output.log
2022-02-01 15:08:26,857 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json wandb-metadata.json
2022-02-01 15:08:26,869 INFO SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py code/run_speech_recognition_ctc.py
2022-02-01 15:08:26,869 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 6
2022-02-01 15:08:26,869 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:26,881 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,881 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 6
2022-02-01 15:08:26,882 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,882 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 6
2022-02-01 15:08:26,882 INFO SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 15:08:26,984 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,984 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,086 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,087 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,189 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,189 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,291 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,292 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,394 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,394 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,497 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,497 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,545 INFO Thread-13 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml
2022-02-01 15:08:27,551 INFO Thread-14 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt
2022-02-01 15:08:27,553 INFO Thread-16 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:27,565 INFO Thread-15 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:27,599 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,600 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,702 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,702 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,766 INFO Thread-7 :3248864 [sender.py:transition_state():387] send defer: 7
2022-02-01 15:08:27,767 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:27,767 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 7
2022-02-01 15:08:27,767 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:27,767 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 7
2022-02-01 15:08:27,805 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:28,272 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 8
2022-02-01 15:08:28,272 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:28,273 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:28,273 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 8
2022-02-01 15:08:28,274 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:28,274 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 8
2022-02-01 15:08:28,274 INFO SenderThread:3248864 [sender.py:transition_state():387] send defer: 9
2022-02-01 15:08:28,275 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:28,275 DEBUG SenderThread:3248864 [sender.py:send():234] send: final
2022-02-01 15:08:28,275 INFO HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 9
2022-02-01 15:08:28,276 DEBUG SenderThread:3248864 [sender.py:send():234] send: footer
2022-02-01 15:08:28,276 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:28,276 INFO SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 9
2022-02-01 15:08:28,375 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:28,375 DEBUG SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:28,375 INFO SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 15:08:28,858 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: get_summary
2022-02-01 15:08:28,859 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: sampled_history
2022-02-01 15:08:28,861 DEBUG HandlerThread:3248864 [handler.py:handle_request():130] handle_request: shutdown
2022-02-01 15:08:28,861 INFO HandlerThread:3248864 [handler.py:finish():731] shutting down handler
2022-02-01 15:08:29,275 INFO WriterThread:3248864 [datastore.py:close():281] close: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb
2022-02-01 15:08:29,856 INFO SenderThread:3248864 [sender.py:finish():1070] shutting down sender
2022-02-01 15:08:29,856 INFO SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 15:08:29,856 INFO SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 15:08:29,859 INFO MainThread:3248864 [internal.py:handle_exit():77] Internal process exited
|