| 2022-01-30 17:51:36,839 INFO MainThread:21293 [internal.py:wandb_internal():87] W&B internal server running at pid: 21293, started at: 2022-01-30 17:51:36.839410 | |
| 2022-01-30 17:51:36,843 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: check_version | |
| 2022-01-30 17:51:36,844 INFO WriterThread:21293 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/run-mz3dbiz4.wandb | |
| 2022-01-30 17:51:36,847 DEBUG SenderThread:21293 [sender.py:send():234] send: header | |
| 2022-01-30 17:51:36,848 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: check_version | |
| 2022-01-30 17:51:36,917 DEBUG SenderThread:21293 [sender.py:send():234] send: run | |
| 2022-01-30 17:51:37,133 INFO SenderThread:21293 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files | |
| 2022-01-30 17:51:37,134 INFO SenderThread:21293 [sender.py:_start_run_threads():804] run started: mz3dbiz4 with start time 1643565096 | |
| 2022-01-30 17:51:37,134 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: run_start | |
| 2022-01-30 17:51:37,134 DEBUG SenderThread:21293 [sender.py:send():234] send: summary | |
| 2022-01-30 17:51:37,136 INFO SenderThread:21293 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-30 17:51:37,144 DEBUG HandlerThread:21293 [meta.py:__init__():40] meta init | |
| 2022-01-30 17:51:37,144 DEBUG HandlerThread:21293 [meta.py:__init__():54] meta init done | |
| 2022-01-30 17:51:37,144 DEBUG HandlerThread:21293 [meta.py:probe():214] probe | |
| 2022-01-30 17:51:37,154 DEBUG HandlerThread:21293 [meta.py:_setup_git():204] setup git | |
| 2022-01-30 17:51:37,190 DEBUG HandlerThread:21293 [meta.py:_setup_git():211] setup git done | |
| 2022-01-30 17:51:37,191 DEBUG HandlerThread:21293 [meta.py:_save_pip():58] save pip | |
| 2022-01-30 17:51:37,193 DEBUG HandlerThread:21293 [meta.py:_save_pip():72] save pip done | |
| 2022-01-30 17:51:37,193 DEBUG HandlerThread:21293 [meta.py:_save_conda():79] save conda | |
| 2022-01-30 17:51:37,717 DEBUG HandlerThread:21293 [meta.py:_save_conda():89] save conda done | |
| 2022-01-30 17:51:37,718 DEBUG HandlerThread:21293 [meta.py:probe():252] probe done | |
| 2022-01-30 17:51:37,726 DEBUG SenderThread:21293 [sender.py:send():234] send: files | |
| 2022-01-30 17:51:37,726 INFO SenderThread:21293 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now | |
| 2022-01-30 17:51:37,736 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-30 17:51:37,737 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-30 17:51:37,890 DEBUG SenderThread:21293 [sender.py:send():234] send: config | |
| 2022-01-30 17:51:37,892 DEBUG SenderThread:21293 [sender.py:send():234] send: metric | |
| 2022-01-30 17:51:37,893 DEBUG SenderThread:21293 [sender.py:send():234] send: metric | |
| 2022-01-30 17:51:37,893 WARNING SenderThread:21293 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen) | |
| 2022-01-30 17:51:38,136 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/conda-environment.yaml | |
| 2022-01-30 17:51:38,137 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-summary.json | |
| 2022-01-30 17:51:38,137 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/requirements.txt | |
| 2022-01-30 17:51:38,138 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:51:38,138 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-metadata.json | |
| 2022-01-30 17:51:38,250 INFO Thread-11 :21293 [upload_job.py:push():137] Uploaded file /tmp/tmp7m8cqty2wandb/246jybel-wandb-metadata.json | |
| 2022-01-30 17:51:40,135 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:51:46,138 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:51:48,140 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:51:50,142 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:51:53,043 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-30 17:51:53,043 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-30 17:51:56,145 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:00,148 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:02,149 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:05,410 DEBUG SenderThread:21293 [sender.py:send():234] send: stats | |
| 2022-01-30 17:52:06,151 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:08,153 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/config.yaml | |
| 2022-01-30 17:52:08,539 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-30 17:52:08,540 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-30 17:52:10,154 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:12,156 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:16,158 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:21,162 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:23,163 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:23,731 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-30 17:52:23,732 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-30 17:52:29,168 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:31,169 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:33,170 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:35,634 DEBUG SenderThread:21293 [sender.py:send():234] send: stats | |
| 2022-01-30 17:52:37,174 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:38,896 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-30 17:52:38,896 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-30 17:52:43,178 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:45,179 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:47,180 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:53,183 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:54,057 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-30 17:52:54,058 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-30 17:52:57,186 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:52:59,187 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:03,190 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:05,866 DEBUG SenderThread:21293 [sender.py:send():234] send: stats | |
| 2022-01-30 17:53:07,192 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:09,194 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:09,222 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-30 17:53:09,223 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-30 17:53:15,197 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:17,198 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:19,200 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:23,992 WARNING MainThread:21293 [internal.py:wandb_internal():146] Internal process interrupt: 1 | |
| 2022-01-30 17:53:25,205 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:26,380 DEBUG SenderThread:21293 [sender.py:send():234] send: telemetry | |
| 2022-01-30 17:53:26,380 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:26,380 DEBUG SenderThread:21293 [sender.py:send():234] send: exit | |
| 2022-01-30 17:53:26,381 INFO SenderThread:21293 [sender.py:send_exit():366] handling exit code: 255 | |
| 2022-01-30 17:53:26,381 INFO SenderThread:21293 [sender.py:send_exit():368] handling runtime: 109 | |
| 2022-01-30 17:53:26,381 INFO SenderThread:21293 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-30 17:53:26,382 INFO SenderThread:21293 [sender.py:send_exit():374] send defer | |
| 2022-01-30 17:53:26,382 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:26,383 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:26,383 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 0 | |
| 2022-01-30 17:53:26,384 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:26,384 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 0 | |
| 2022-01-30 17:53:26,384 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 1 | |
| 2022-01-30 17:53:26,384 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:26,385 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 1 | |
| 2022-01-30 17:53:26,424 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:26,424 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 1 | |
| 2022-01-30 17:53:26,424 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 2 | |
| 2022-01-30 17:53:26,425 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:26,425 DEBUG SenderThread:21293 [sender.py:send():234] send: stats | |
| 2022-01-30 17:53:26,426 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 2 | |
| 2022-01-30 17:53:26,427 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:26,427 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 2 | |
| 2022-01-30 17:53:26,427 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 3 | |
| 2022-01-30 17:53:26,427 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:26,428 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 3 | |
| 2022-01-30 17:53:26,428 DEBUG SenderThread:21293 [sender.py:send():234] send: summary | |
| 2022-01-30 17:53:26,429 INFO SenderThread:21293 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-30 17:53:26,429 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:26,429 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 3 | |
| 2022-01-30 17:53:26,430 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 4 | |
| 2022-01-30 17:53:26,430 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:26,430 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 4 | |
| 2022-01-30 17:53:26,431 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:26,431 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 4 | |
| 2022-01-30 17:53:26,485 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:26,670 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 5 | |
| 2022-01-30 17:53:26,670 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:26,670 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:26,671 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 5 | |
| 2022-01-30 17:53:26,671 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:26,671 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 5 | |
| 2022-01-30 17:53:26,671 INFO SenderThread:21293 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-01-30 17:53:26,772 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:27,207 INFO SenderThread:21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/config.yaml | |
| 2022-01-30 17:53:27,207 INFO SenderThread:21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:27,208 INFO SenderThread:21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-summary.json | |
| 2022-01-30 17:53:27,208 INFO SenderThread:21293 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files | |
| 2022-01-30 17:53:27,209 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log output.log | |
| 2022-01-30 17:53:27,209 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-metadata.json wandb-metadata.json | |
| 2022-01-30 17:53:27,209 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/config.yaml config.yaml | |
| 2022-01-30 17:53:27,210 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/conda-environment.yaml conda-environment.yaml | |
| 2022-01-30 17:53:27,222 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-summary.json wandb-summary.json | |
| 2022-01-30 17:53:27,223 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/requirements.txt requirements.txt | |
| 2022-01-30 17:53:27,224 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 6 | |
| 2022-01-30 17:53:27,224 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:27,233 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:27,233 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 6 | |
| 2022-01-30 17:53:27,241 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:27,242 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 6 | |
| 2022-01-30 17:53:27,242 INFO SenderThread:21293 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-01-30 17:53:27,329 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:27,329 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:27,436 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:27,440 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:27,543 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:27,543 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:27,646 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:27,646 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:27,738 INFO Thread-12 :21293 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log | |
| 2022-01-30 17:53:27,746 INFO Thread-14 :21293 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-summary.json | |
| 2022-01-30 17:53:27,748 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:27,748 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:27,761 INFO Thread-13 :21293 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/config.yaml | |
| 2022-01-30 17:53:27,775 INFO Thread-15 :21293 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/requirements.txt | |
| 2022-01-30 17:53:27,851 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:27,851 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:27,954 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:27,954 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:27,976 INFO Thread-7 :21293 [sender.py:transition_state():387] send defer: 7 | |
| 2022-01-30 17:53:27,977 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:27,977 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 7 | |
| 2022-01-30 17:53:27,977 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:27,977 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 7 | |
| 2022-01-30 17:53:28,057 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:28,973 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 8 | |
| 2022-01-30 17:53:28,974 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:28,974 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:28,975 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 8 | |
| 2022-01-30 17:53:28,975 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:28,975 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 8 | |
| 2022-01-30 17:53:28,975 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 9 | |
| 2022-01-30 17:53:28,976 DEBUG SenderThread:21293 [sender.py:send():234] send: final | |
| 2022-01-30 17:53:28,977 DEBUG SenderThread:21293 [sender.py:send():234] send: footer | |
| 2022-01-30 17:53:28,977 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:53:28,977 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 9 | |
| 2022-01-30 17:53:28,977 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:53:28,977 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 9 | |
| 2022-01-30 17:53:29,077 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:53:29,077 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:53:29,078 INFO SenderThread:21293 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-01-30 17:53:29,426 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: get_summary | |
| 2022-01-30 17:53:29,427 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: sampled_history | |
| 2022-01-30 17:53:29,429 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: shutdown | |
| 2022-01-30 17:53:29,429 INFO HandlerThread:21293 [handler.py:finish():731] shutting down handler | |
| 2022-01-30 17:53:29,977 INFO WriterThread:21293 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/run-mz3dbiz4.wandb | |
| 2022-01-30 17:53:30,424 INFO SenderThread:21293 [sender.py:finish():1070] shutting down sender | |
| 2022-01-30 17:53:30,425 INFO SenderThread:21293 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-01-30 17:53:30,425 INFO SenderThread:21293 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-01-30 17:53:30,428 INFO MainThread:21293 [internal.py:handle_exit():77] Internal process exited | |