2022-01-30 17:54:18,493 INFO MainThread:22568 [internal.py:wandb_internal():87] W&B internal server running at pid: 22568, started at: 2022-01-30 17:54:18.493348 2022-01-30 17:54:18,496 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: check_version 2022-01-30 17:54:18,497 INFO WriterThread:22568 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/run-3n44neww.wandb 2022-01-30 17:54:18,498 DEBUG SenderThread:22568 [sender.py:send():234] send: header 2022-01-30 17:54:18,498 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: check_version 2022-01-30 17:54:18,588 DEBUG SenderThread:22568 [sender.py:send():234] send: run 2022-01-30 17:54:18,778 INFO SenderThread:22568 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files 2022-01-30 17:54:18,779 INFO SenderThread:22568 [sender.py:_start_run_threads():804] run started: 3n44neww with start time 1643565257 2022-01-30 17:54:18,779 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:54:18,779 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:54:18,781 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: run_start 2022-01-30 17:54:18,803 DEBUG HandlerThread:22568 [meta.py:__init__():40] meta init 2022-01-30 17:54:18,803 DEBUG HandlerThread:22568 [meta.py:__init__():54] meta init done 2022-01-30 17:54:18,803 DEBUG HandlerThread:22568 [meta.py:probe():214] probe 2022-01-30 17:54:18,814 DEBUG HandlerThread:22568 [meta.py:_setup_git():204] setup git 2022-01-30 17:54:18,844 DEBUG HandlerThread:22568 [meta.py:_setup_git():211] setup git done 2022-01-30 17:54:18,844 DEBUG HandlerThread:22568 [meta.py:_save_pip():58] save pip 2022-01-30 17:54:18,845 DEBUG HandlerThread:22568 [meta.py:_save_pip():72] save pip done 2022-01-30 17:54:18,845 DEBUG HandlerThread:22568 [meta.py:_save_conda():79] save conda 2022-01-30 17:54:19,393 DEBUG HandlerThread:22568 [meta.py:_save_conda():89] save conda done 2022-01-30 17:54:19,394 DEBUG HandlerThread:22568 [meta.py:probe():252] probe done 2022-01-30 17:54:19,402 DEBUG SenderThread:22568 [sender.py:send():234] send: files 2022-01-30 17:54:19,402 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now 2022-01-30 17:54:19,413 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status 2022-01-30 17:54:19,414 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: stop_status 2022-01-30 17:54:19,575 DEBUG SenderThread:22568 [sender.py:send():234] send: config 2022-01-30 17:54:19,578 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:19,578 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:19,578 WARNING SenderThread:22568 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen) 2022-01-30 17:54:19,782 INFO Thread-8 :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:54:19,782 INFO Thread-8 :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:19,783 INFO Thread-8 :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/requirements.txt 2022-01-30 17:54:19,783 INFO Thread-8 :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/conda-environment.yaml 2022-01-30 17:54:19,783 INFO Thread-8 :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-metadata.json 2022-01-30 17:54:19,939 INFO Thread-11 :22568 [upload_job.py:push():137] Uploaded file /tmp/tmpzu6sqwh8wandb/1uonnopl-wandb-metadata.json 2022-01-30 17:54:21,780 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:27,784 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:29,786 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:31,787 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:34,745 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status 2022-01-30 17:54:34,745 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: stop_status 2022-01-30 17:54:37,791 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:39,019 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:39,020 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:39,020 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:39,021 DEBUG SenderThread:22568 [sender.py:send():234] send: history 2022-01-30 17:54:39,021 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:54:39,022 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:54:39,792 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:54:39,793 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:41,854 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:41,854 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:41,855 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:41,855 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:41,855 DEBUG SenderThread:22568 [sender.py:send():234] send: metric 2022-01-30 17:54:41,855 DEBUG SenderThread:22568 [sender.py:send():234] send: history 2022-01-30 17:54:41,856 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:54:41,861 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:54:42,794 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:54:43,795 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:45,797 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:47,048 DEBUG SenderThread:22568 [sender.py:send():234] send: stats 2022-01-30 17:54:49,799 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:49,800 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/config.yaml 2022-01-30 17:54:49,989 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status 2022-01-30 17:54:49,989 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: stop_status 2022-01-30 17:54:53,802 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:56,803 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:54:59,567 DEBUG SenderThread:22568 [sender.py:send():234] send: history 2022-01-30 17:54:59,568 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:54:59,571 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:54:59,805 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:55:01,807 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:02,327 DEBUG SenderThread:22568 [sender.py:send():234] send: history 2022-01-30 17:55:02,328 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:55:02,329 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:55:02,807 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:55:02,808 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:03,808 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:04,809 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:05,223 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status 2022-01-30 17:55:05,224 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: stop_status 2022-01-30 17:55:06,810 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:10,813 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:14,815 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:17,371 DEBUG SenderThread:22568 [sender.py:send():234] send: stats 2022-01-30 17:55:18,522 DEBUG SenderThread:22568 [sender.py:send():234] send: history 2022-01-30 17:55:18,523 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:55:18,524 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:55:18,817 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:55:18,818 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:19,818 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:20,385 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status 2022-01-30 17:55:20,386 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: stop_status 2022-01-30 17:55:20,819 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:21,292 DEBUG SenderThread:22568 [sender.py:send():234] send: history 2022-01-30 17:55:21,293 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:55:21,294 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:55:21,819 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:55:22,820 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:26,823 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:32,827 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:34,828 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:35,544 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status 2022-01-30 17:55:35,544 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: stop_status 2022-01-30 17:55:36,830 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:41,459 DEBUG SenderThread:22568 [sender.py:send():234] send: history 2022-01-30 17:55:41,459 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:55:41,460 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:55:41,833 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:55:42,834 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:44,141 DEBUG SenderThread:22568 [sender.py:send():234] send: history 2022-01-30 17:55:44,142 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:55:44,143 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:55:44,835 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:55:44,836 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:46,836 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:47,657 DEBUG SenderThread:22568 [sender.py:send():234] send: stats 2022-01-30 17:55:48,838 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:50,808 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status 2022-01-30 17:55:50,809 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: stop_status 2022-01-30 17:55:50,839 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:51,470 WARNING MainThread:22568 [internal.py:wandb_internal():146] Internal process interrupt: 1 2022-01-30 17:55:52,841 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:54,071 DEBUG SenderThread:22568 [sender.py:send():234] send: telemetry 2022-01-30 17:55:54,072 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:54,072 DEBUG SenderThread:22568 [sender.py:send():234] send: exit 2022-01-30 17:55:54,073 INFO SenderThread:22568 [sender.py:send_exit():366] handling exit code: 255 2022-01-30 17:55:54,073 INFO SenderThread:22568 [sender.py:send_exit():368] handling runtime: 95 2022-01-30 17:55:54,074 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:55:54,074 INFO SenderThread:22568 [sender.py:send_exit():374] send defer 2022-01-30 17:55:54,075 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:54,075 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:54,076 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 0 2022-01-30 17:55:54,077 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:54,077 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 0 2022-01-30 17:55:54,077 INFO SenderThread:22568 [sender.py:transition_state():387] send defer: 1 2022-01-30 17:55:54,077 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:54,078 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 1 2022-01-30 17:55:54,104 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:54,104 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 1 2022-01-30 17:55:54,104 INFO SenderThread:22568 [sender.py:transition_state():387] send defer: 2 2022-01-30 17:55:54,105 DEBUG SenderThread:22568 [sender.py:send():234] send: stats 2022-01-30 17:55:54,106 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:54,106 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 2 2022-01-30 17:55:54,106 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:54,106 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 2 2022-01-30 17:55:54,106 INFO SenderThread:22568 [sender.py:transition_state():387] send defer: 3 2022-01-30 17:55:54,107 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:54,107 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 3 2022-01-30 17:55:54,107 DEBUG SenderThread:22568 [sender.py:send():234] send: summary 2022-01-30 17:55:54,108 INFO SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 17:55:54,108 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:54,108 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 3 2022-01-30 17:55:54,109 INFO SenderThread:22568 [sender.py:transition_state():387] send defer: 4 2022-01-30 17:55:54,109 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:54,109 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 4 2022-01-30 17:55:54,109 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:54,109 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 4 2022-01-30 17:55:54,178 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:54,344 INFO SenderThread:22568 [sender.py:transition_state():387] send defer: 5 2022-01-30 17:55:54,344 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:54,346 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:54,346 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 5 2022-01-30 17:55:54,346 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:54,346 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 5 2022-01-30 17:55:54,346 INFO SenderThread:22568 [dir_watcher.py:finish():283] shutting down directory watcher 2022-01-30 17:55:54,448 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:54,842 INFO Thread-8 :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:55:54,844 INFO SenderThread:22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:54,844 INFO SenderThread:22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/config.yaml 2022-01-30 17:55:54,845 INFO SenderThread:22568 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files 2022-01-30 17:55:54,845 INFO SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log output.log 2022-01-30 17:55:54,845 INFO SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-metadata.json wandb-metadata.json 2022-01-30 17:55:54,846 INFO SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/config.yaml config.yaml 2022-01-30 17:55:54,846 INFO SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/conda-environment.yaml conda-environment.yaml 2022-01-30 17:55:54,847 INFO SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json wandb-summary.json 2022-01-30 17:55:54,860 INFO SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/requirements.txt requirements.txt 2022-01-30 17:55:54,868 INFO SenderThread:22568 [sender.py:transition_state():387] send defer: 6 2022-01-30 17:55:54,869 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:54,878 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:54,879 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 6 2022-01-30 17:55:54,879 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:54,880 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 6 2022-01-30 17:55:54,880 INFO SenderThread:22568 [file_pusher.py:finish():177] shutting down file pusher 2022-01-30 17:55:54,973 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:54,974 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:55,077 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:55,078 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:55,181 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:55,182 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:55,286 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:55,286 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:55,368 INFO Thread-13 :22568 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/config.yaml 2022-01-30 17:55:55,370 INFO Thread-14 :22568 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json 2022-01-30 17:55:55,373 INFO Thread-15 :22568 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/requirements.txt 2022-01-30 17:55:55,390 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:55,390 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:55,400 INFO Thread-12 :22568 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log 2022-01-30 17:55:55,493 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:55,493 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:55,596 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:55,597 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:55,601 INFO Thread-7 :22568 [sender.py:transition_state():387] send defer: 7 2022-01-30 17:55:55,602 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:55,602 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 7 2022-01-30 17:55:55,602 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:55,602 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 7 2022-01-30 17:55:55,699 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:56,187 INFO SenderThread:22568 [sender.py:transition_state():387] send defer: 8 2022-01-30 17:55:56,187 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:56,188 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:56,189 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 8 2022-01-30 17:55:56,189 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:56,189 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 8 2022-01-30 17:55:56,189 INFO SenderThread:22568 [sender.py:transition_state():387] send defer: 9 2022-01-30 17:55:56,191 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer 2022-01-30 17:55:56,191 INFO HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 9 2022-01-30 17:55:56,191 DEBUG SenderThread:22568 [sender.py:send():234] send: final 2022-01-30 17:55:56,191 DEBUG SenderThread:22568 [sender.py:send():234] send: footer 2022-01-30 17:55:56,192 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: defer 2022-01-30 17:55:56,192 INFO SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 9 2022-01-30 17:55:56,290 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 17:55:56,291 DEBUG SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 17:55:56,291 INFO SenderThread:22568 [file_pusher.py:join():182] waiting for file pusher 2022-01-30 17:55:56,600 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: get_summary 2022-01-30 17:55:56,602 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: sampled_history 2022-01-30 17:55:56,604 DEBUG HandlerThread:22568 [handler.py:handle_request():130] handle_request: shutdown 2022-01-30 17:55:56,604 INFO HandlerThread:22568 [handler.py:finish():731] shutting down handler 2022-01-30 17:55:57,191 INFO WriterThread:22568 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/run-3n44neww.wandb 2022-01-30 17:55:57,598 INFO SenderThread:22568 [sender.py:finish():1070] shutting down sender 2022-01-30 17:55:57,598 INFO SenderThread:22568 [file_pusher.py:finish():177] shutting down file pusher 2022-01-30 17:55:57,598 INFO SenderThread:22568 [file_pusher.py:join():182] waiting for file pusher 2022-01-30 17:55:57,602 INFO MainThread:22568 [internal.py:handle_exit():77] Internal process exited