2022-02-01 10:30:05,624 INFO MainThread:61045 [internal.py:wandb_internal():87] W&B internal server running at pid: 61045, started at: 2022-02-01 10:30:05.624191 2022-02-01 10:30:05,628 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: check_version 2022-02-01 10:30:05,628 INFO WriterThread:61045 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/run-1yfj7vwy.wandb 2022-02-01 10:30:05,632 DEBUG SenderThread:61045 [sender.py:send():234] send: header 2022-02-01 10:30:05,632 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: check_version 2022-02-01 10:30:05,706 DEBUG SenderThread:61045 [sender.py:send():234] send: run 2022-02-01 10:30:05,896 INFO SenderThread:61045 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files 2022-02-01 10:30:05,896 INFO SenderThread:61045 [sender.py:_start_run_threads():804] run started: 1yfj7vwy with start time 1643711404 2022-02-01 10:30:05,896 DEBUG SenderThread:61045 [sender.py:send():234] send: summary 2022-02-01 10:30:05,897 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-02-01 10:30:05,898 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: run_start 2022-02-01 10:30:05,908 DEBUG HandlerThread:61045 [meta.py:__init__():40] meta init 2022-02-01 10:30:05,909 DEBUG HandlerThread:61045 [meta.py:__init__():54] meta init done 2022-02-01 10:30:05,909 DEBUG HandlerThread:61045 [meta.py:probe():214] probe 2022-02-01 10:30:05,917 DEBUG HandlerThread:61045 [meta.py:_setup_git():204] setup git 2022-02-01 10:30:05,952 DEBUG HandlerThread:61045 [meta.py:_setup_git():211] setup git done 2022-02-01 10:30:05,953 DEBUG HandlerThread:61045 [meta.py:_save_pip():58] save pip 2022-02-01 10:30:05,954 DEBUG HandlerThread:61045 [meta.py:_save_pip():72] save pip done 2022-02-01 10:30:05,954 DEBUG HandlerThread:61045 [meta.py:_save_conda():79] save conda 2022-02-01 10:30:06,494 DEBUG HandlerThread:61045 [meta.py:_save_conda():89] save conda done 2022-02-01 10:30:06,495 DEBUG HandlerThread:61045 [meta.py:probe():252] probe done 2022-02-01 10:30:06,500 DEBUG SenderThread:61045 [sender.py:send():234] send: files 2022-02-01 10:30:06,501 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now 2022-02-01 10:30:06,511 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:30:06,512 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:30:06,673 DEBUG SenderThread:61045 [sender.py:send():234] send: config 2022-02-01 10:30:06,675 DEBUG SenderThread:61045 [sender.py:send():234] send: metric 2022-02-01 10:30:06,676 DEBUG SenderThread:61045 [sender.py:send():234] send: metric 2022-02-01 10:30:06,676 WARNING SenderThread:61045 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen) 2022-02-01 10:30:06,676 DEBUG SenderThread:61045 [sender.py:send():234] send: metric 2022-02-01 10:30:06,676 DEBUG SenderThread:61045 [sender.py:send():234] send: metric 2022-02-01 10:30:06,676 DEBUG SenderThread:61045 [sender.py:send():234] send: metric 2022-02-01 10:30:06,677 DEBUG SenderThread:61045 [sender.py:send():234] send: metric 2022-02-01 10:30:06,677 DEBUG SenderThread:61045 [sender.py:send():234] send: metric 2022-02-01 10:30:06,677 DEBUG SenderThread:61045 [sender.py:send():234] send: history 2022-02-01 10:30:06,677 DEBUG SenderThread:61045 [sender.py:send():234] send: summary 2022-02-01 10:30:06,678 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-02-01 10:30:06,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/conda-environment.yaml 2022-02-01 10:30:06,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:30:06,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json 2022-02-01 10:30:06,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-metadata.json 2022-02-01 10:30:06,899 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/requirements.txt 2022-02-01 10:30:07,139 INFO Thread-11 :61045 [upload_job.py:push():137] Uploaded file /tmp/tmpzeoerhazwandb/25qe2qjl-wandb-metadata.json 2022-02-01 10:30:08,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:30:10,899 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:30:21,675 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:30:21,676 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:30:34,180 DEBUG SenderThread:61045 [sender.py:send():234] send: stats 2022-02-01 10:30:36,839 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:30:36,839 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:30:36,921 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/config.yaml 2022-02-01 10:30:52,000 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:30:52,001 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:31:04,514 DEBUG SenderThread:61045 [sender.py:send():234] send: stats 2022-02-01 10:31:07,162 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:31:07,163 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:31:22,319 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:31:22,320 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:31:24,949 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:31:30,952 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:31:32,953 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:31:34,785 DEBUG SenderThread:61045 [sender.py:send():234] send: stats 2022-02-01 10:31:34,954 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:31:36,955 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:31:37,488 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:31:37,489 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:31:38,957 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:31:52,642 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:31:52,643 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:31:52,963 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:32:05,018 DEBUG SenderThread:61045 [sender.py:send():234] send: stats 2022-02-01 10:32:07,832 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:32:07,833 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:32:23,003 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:32:23,003 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:32:35,259 DEBUG SenderThread:61045 [sender.py:send():234] send: stats 2022-02-01 10:32:38,164 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status 2022-02-01 10:32:38,165 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status 2022-02-01 10:32:42,989 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:32:44,990 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:32:48,993 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:32:52,987 DEBUG SenderThread:61045 [sender.py:send():234] send: telemetry 2022-02-01 10:32:52,988 DEBUG SenderThread:61045 [sender.py:send():234] send: exit 2022-02-01 10:32:52,988 INFO SenderThread:61045 [sender.py:send_exit():366] handling exit code: 0 2022-02-01 10:32:52,988 INFO SenderThread:61045 [sender.py:send_exit():368] handling runtime: 167 2022-02-01 10:32:52,989 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:52,990 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-02-01 10:32:52,990 INFO SenderThread:61045 [sender.py:send_exit():374] send defer 2022-02-01 10:32:52,990 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:52,991 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:52,992 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 0 2022-02-01 10:32:52,992 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:52,992 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 0 2022-02-01 10:32:52,992 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 1 2022-02-01 10:32:52,993 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:52,993 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 1 2022-02-01 10:32:52,996 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:32:52,996 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json 2022-02-01 10:32:53,088 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:53,088 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 1 2022-02-01 10:32:53,088 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 2 2022-02-01 10:32:53,088 DEBUG SenderThread:61045 [sender.py:send():234] send: stats 2022-02-01 10:32:53,089 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:53,089 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 2 2022-02-01 10:32:53,090 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:53,090 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 2 2022-02-01 10:32:53,090 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 3 2022-02-01 10:32:53,090 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:53,090 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 3 2022-02-01 10:32:53,091 DEBUG SenderThread:61045 [sender.py:send():234] send: summary 2022-02-01 10:32:53,092 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-02-01 10:32:53,092 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:53,092 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 3 2022-02-01 10:32:53,092 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 4 2022-02-01 10:32:53,093 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:53,093 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 4 2022-02-01 10:32:53,093 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:53,093 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 4 2022-02-01 10:32:53,103 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:53,344 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 5 2022-02-01 10:32:53,344 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:53,345 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:53,345 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 5 2022-02-01 10:32:53,345 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:53,345 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 5 2022-02-01 10:32:53,345 INFO SenderThread:61045 [dir_watcher.py:finish():283] shutting down directory watcher 2022-02-01 10:32:53,448 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:53,997 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/config.yaml 2022-02-01 10:32:53,998 INFO SenderThread:61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json 2022-02-01 10:32:53,999 INFO SenderThread:61045 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files 2022-02-01 10:32:53,999 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log output.log 2022-02-01 10:32:54,000 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-metadata.json wandb-metadata.json 2022-02-01 10:32:54,001 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/config.yaml config.yaml 2022-02-01 10:32:54,007 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/conda-environment.yaml conda-environment.yaml 2022-02-01 10:32:54,014 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json wandb-summary.json 2022-02-01 10:32:54,015 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/requirements.txt requirements.txt 2022-02-01 10:32:54,019 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 6 2022-02-01 10:32:54,020 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:54,026 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:54,026 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 6 2022-02-01 10:32:54,026 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:54,027 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 6 2022-02-01 10:32:54,027 INFO SenderThread:61045 [file_pusher.py:finish():177] shutting down file pusher 2022-02-01 10:32:54,122 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:54,123 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:54,225 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:54,226 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:54,328 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:54,329 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:54,431 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:54,432 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:54,524 INFO Thread-14 :61045 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json 2022-02-01 10:32:54,531 INFO Thread-12 :61045 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log 2022-02-01 10:32:54,534 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:54,534 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:54,542 INFO Thread-15 :61045 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/requirements.txt 2022-02-01 10:32:54,556 INFO Thread-13 :61045 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/config.yaml 2022-02-01 10:32:54,636 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:54,637 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:54,739 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:54,739 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:54,757 INFO Thread-7 :61045 [sender.py:transition_state():387] send defer: 7 2022-02-01 10:32:54,758 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:54,759 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 7 2022-02-01 10:32:54,759 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:54,759 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 7 2022-02-01 10:32:54,842 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:55,205 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 8 2022-02-01 10:32:55,206 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:55,206 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:55,207 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 8 2022-02-01 10:32:55,208 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:55,208 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 8 2022-02-01 10:32:55,208 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 9 2022-02-01 10:32:55,209 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer 2022-02-01 10:32:55,210 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 9 2022-02-01 10:32:55,210 DEBUG SenderThread:61045 [sender.py:send():234] send: final 2022-02-01 10:32:55,210 DEBUG SenderThread:61045 [sender.py:send():234] send: footer 2022-02-01 10:32:55,210 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer 2022-02-01 10:32:55,210 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 9 2022-02-01 10:32:55,309 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit 2022-02-01 10:32:55,310 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit 2022-02-01 10:32:55,310 INFO SenderThread:61045 [file_pusher.py:join():182] waiting for file pusher 2022-02-01 10:32:55,613 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: get_summary 2022-02-01 10:32:55,615 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: sampled_history 2022-02-01 10:32:55,617 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: shutdown 2022-02-01 10:32:55,617 INFO HandlerThread:61045 [handler.py:finish():731] shutting down handler 2022-02-01 10:32:56,210 INFO WriterThread:61045 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/run-1yfj7vwy.wandb 2022-02-01 10:32:56,611 INFO SenderThread:61045 [sender.py:finish():1070] shutting down sender 2022-02-01 10:32:56,611 INFO SenderThread:61045 [file_pusher.py:finish():177] shutting down file pusher 2022-02-01 10:32:56,611 INFO SenderThread:61045 [file_pusher.py:join():182] waiting for file pusher 2022-02-01 10:32:56,615 INFO MainThread:61045 [internal.py:handle_exit():77] Internal process exited