2022-01-30 16:32:48,452 INFO MainThread:52211 [internal.py:wandb_internal():87] W&B internal server running at pid: 52211, started at: 2022-01-30 16:32:48.452114 2022-01-30 16:32:48,456 INFO WriterThread:52211 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/run-20w0bl60.wandb 2022-01-30 16:32:48,457 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: check_version 2022-01-30 16:32:48,460 DEBUG SenderThread:52211 [sender.py:send():234] send: header 2022-01-30 16:32:48,460 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: check_version 2022-01-30 16:32:48,657 DEBUG SenderThread:52211 [sender.py:send():234] send: run 2022-01-30 16:32:48,873 INFO SenderThread:52211 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files 2022-01-30 16:32:48,873 INFO SenderThread:52211 [sender.py:_start_run_threads():804] run started: 20w0bl60 with start time 1643560367 2022-01-30 16:32:48,873 DEBUG SenderThread:52211 [sender.py:send():234] send: summary 2022-01-30 16:32:48,874 INFO SenderThread:52211 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 16:32:48,876 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: run_start 2022-01-30 16:32:48,884 DEBUG HandlerThread:52211 [meta.py:__init__():40] meta init 2022-01-30 16:32:48,885 DEBUG HandlerThread:52211 [meta.py:__init__():54] meta init done 2022-01-30 16:32:48,885 DEBUG HandlerThread:52211 [meta.py:probe():214] probe 2022-01-30 16:32:48,894 DEBUG HandlerThread:52211 [meta.py:_setup_git():204] setup git 2022-01-30 16:32:48,923 DEBUG HandlerThread:52211 [meta.py:_setup_git():211] setup git done 2022-01-30 16:32:48,923 DEBUG HandlerThread:52211 [meta.py:_save_pip():58] save pip 2022-01-30 16:32:48,924 DEBUG HandlerThread:52211 [meta.py:_save_pip():72] save pip done 2022-01-30 16:32:48,924 DEBUG HandlerThread:52211 [meta.py:_save_conda():79] save conda 2022-01-30 16:32:49,449 DEBUG HandlerThread:52211 [meta.py:_save_conda():89] save conda done 2022-01-30 16:32:49,450 DEBUG HandlerThread:52211 [meta.py:probe():252] probe done 2022-01-30 16:32:49,458 DEBUG SenderThread:52211 [sender.py:send():234] send: files 2022-01-30 16:32:49,459 INFO SenderThread:52211 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now 2022-01-30 16:32:49,475 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: stop_status 2022-01-30 16:32:49,476 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: stop_status 2022-01-30 16:32:49,635 DEBUG SenderThread:52211 [sender.py:send():234] send: config 2022-01-30 16:32:49,637 DEBUG SenderThread:52211 [sender.py:send():234] send: metric 2022-01-30 16:32:49,639 DEBUG SenderThread:52211 [sender.py:send():234] send: metric 2022-01-30 16:32:49,639 WARNING SenderThread:52211 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen) 2022-01-30 16:32:49,876 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json 2022-01-30 16:32:49,877 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log 2022-01-30 16:32:49,877 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-metadata.json 2022-01-30 16:32:49,878 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/requirements.txt 2022-01-30 16:32:49,878 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/conda-environment.yaml 2022-01-30 16:32:50,007 INFO Thread-11 :52211 [upload_job.py:push():137] Uploaded file /tmp/tmphz4zp49fwandb/20m3u1mt-wandb-metadata.json 2022-01-30 16:32:51,875 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log 2022-01-30 16:32:55,878 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log 2022-01-30 16:32:56,702 DEBUG SenderThread:52211 [sender.py:send():234] send: telemetry 2022-01-30 16:32:56,703 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:56,703 DEBUG SenderThread:52211 [sender.py:send():234] send: exit 2022-01-30 16:32:56,703 INFO SenderThread:52211 [sender.py:send_exit():366] handling exit code: 1 2022-01-30 16:32:56,704 INFO SenderThread:52211 [sender.py:send_exit():368] handling runtime: 7 2022-01-30 16:32:56,704 INFO SenderThread:52211 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 16:32:56,705 INFO SenderThread:52211 [sender.py:send_exit():374] send defer 2022-01-30 16:32:56,705 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:56,706 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:56,707 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 0 2022-01-30 16:32:56,707 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:56,707 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 0 2022-01-30 16:32:56,707 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 1 2022-01-30 16:32:56,708 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:56,708 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 1 2022-01-30 16:32:56,780 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:56,781 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 1 2022-01-30 16:32:56,781 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 2 2022-01-30 16:32:56,781 DEBUG SenderThread:52211 [sender.py:send():234] send: stats 2022-01-30 16:32:56,782 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:56,782 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 2 2022-01-30 16:32:56,783 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:56,783 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 2 2022-01-30 16:32:56,783 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 3 2022-01-30 16:32:56,784 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:56,784 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 3 2022-01-30 16:32:56,784 DEBUG SenderThread:52211 [sender.py:send():234] send: summary 2022-01-30 16:32:56,785 INFO SenderThread:52211 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-30 16:32:56,785 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:56,785 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 3 2022-01-30 16:32:56,785 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 4 2022-01-30 16:32:56,786 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:56,786 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 4 2022-01-30 16:32:56,787 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:56,787 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 4 2022-01-30 16:32:56,808 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:56,879 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log 2022-01-30 16:32:56,879 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json 2022-01-30 16:32:57,074 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 5 2022-01-30 16:32:57,074 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:57,075 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:57,076 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 5 2022-01-30 16:32:57,076 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:57,076 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 5 2022-01-30 16:32:57,076 INFO SenderThread:52211 [dir_watcher.py:finish():283] shutting down directory watcher 2022-01-30 16:32:57,177 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:57,879 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/config.yaml 2022-01-30 16:32:57,881 INFO SenderThread:52211 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files 2022-01-30 16:32:57,881 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log output.log 2022-01-30 16:32:57,882 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-metadata.json wandb-metadata.json 2022-01-30 16:32:57,882 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/config.yaml config.yaml 2022-01-30 16:32:57,888 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/conda-environment.yaml conda-environment.yaml 2022-01-30 16:32:57,896 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json wandb-summary.json 2022-01-30 16:32:57,897 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/requirements.txt requirements.txt 2022-01-30 16:32:57,904 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 6 2022-01-30 16:32:57,905 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:57,907 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:57,912 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 6 2022-01-30 16:32:57,913 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:57,915 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 6 2022-01-30 16:32:57,915 INFO SenderThread:52211 [file_pusher.py:finish():177] shutting down file pusher 2022-01-30 16:32:58,007 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:58,008 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:58,111 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:58,111 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:58,214 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:58,214 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:58,317 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:58,318 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:58,419 INFO Thread-13 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/config.yaml 2022-01-30 16:32:58,424 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:58,427 INFO Thread-12 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log 2022-01-30 16:32:58,428 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:58,471 INFO Thread-15 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/requirements.txt 2022-01-30 16:32:58,515 INFO Thread-14 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json 2022-01-30 16:32:58,531 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:58,532 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:58,634 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:58,634 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:58,716 INFO Thread-7 :52211 [sender.py:transition_state():387] send defer: 7 2022-01-30 16:32:58,717 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:58,717 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 7 2022-01-30 16:32:58,718 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:58,718 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 7 2022-01-30 16:32:58,736 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:58,862 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 8 2022-01-30 16:32:58,862 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:58,863 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:58,864 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 8 2022-01-30 16:32:58,864 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:58,864 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 8 2022-01-30 16:32:58,864 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 9 2022-01-30 16:32:58,866 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer 2022-01-30 16:32:58,866 DEBUG SenderThread:52211 [sender.py:send():234] send: final 2022-01-30 16:32:58,866 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 9 2022-01-30 16:32:58,866 DEBUG SenderThread:52211 [sender.py:send():234] send: footer 2022-01-30 16:32:58,867 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer 2022-01-30 16:32:58,867 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 9 2022-01-30 16:32:58,965 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-30 16:32:58,966 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit 2022-01-30 16:32:58,967 INFO SenderThread:52211 [file_pusher.py:join():182] waiting for file pusher 2022-01-30 16:32:59,277 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: get_summary 2022-01-30 16:32:59,279 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: sampled_history 2022-01-30 16:32:59,280 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: shutdown 2022-01-30 16:32:59,280 INFO HandlerThread:52211 [handler.py:finish():731] shutting down handler 2022-01-30 16:32:59,866 INFO WriterThread:52211 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/run-20w0bl60.wandb 2022-01-30 16:33:00,275 INFO SenderThread:52211 [sender.py:finish():1070] shutting down sender 2022-01-30 16:33:00,275 INFO SenderThread:52211 [file_pusher.py:finish():177] shutting down file pusher 2022-01-30 16:33:00,275 INFO SenderThread:52211 [file_pusher.py:join():182] waiting for file pusher 2022-01-30 16:33:00,279 INFO MainThread:52211 [internal.py:handle_exit():77] Internal process exited