2022-02-01 10:30:05,624 INFO    MainThread:61045 [internal.py:wandb_internal():87] W&B internal server running at pid: 61045, started at: 2022-02-01 10:30:05.624191
2022-02-01 10:30:05,628 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 10:30:05,628 INFO    WriterThread:61045 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/run-1yfj7vwy.wandb
2022-02-01 10:30:05,632 DEBUG   SenderThread:61045 [sender.py:send():234] send: header
2022-02-01 10:30:05,632 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: check_version
2022-02-01 10:30:05,706 DEBUG   SenderThread:61045 [sender.py:send():234] send: run
2022-02-01 10:30:05,896 INFO    SenderThread:61045 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files
2022-02-01 10:30:05,896 INFO    SenderThread:61045 [sender.py:_start_run_threads():804] run started: 1yfj7vwy with start time 1643711404
2022-02-01 10:30:05,896 DEBUG   SenderThread:61045 [sender.py:send():234] send: summary
2022-02-01 10:30:05,897 INFO    SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 10:30:05,898 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 10:30:05,908 DEBUG   HandlerThread:61045 [meta.py:__init__():40] meta init
2022-02-01 10:30:05,909 DEBUG   HandlerThread:61045 [meta.py:__init__():54] meta init done
2022-02-01 10:30:05,909 DEBUG   HandlerThread:61045 [meta.py:probe():214] probe
2022-02-01 10:30:05,917 DEBUG   HandlerThread:61045 [meta.py:_setup_git():204] setup git
2022-02-01 10:30:05,952 DEBUG   HandlerThread:61045 [meta.py:_setup_git():211] setup git done
2022-02-01 10:30:05,953 DEBUG   HandlerThread:61045 [meta.py:_save_pip():58] save pip
2022-02-01 10:30:05,954 DEBUG   HandlerThread:61045 [meta.py:_save_pip():72] save pip done
2022-02-01 10:30:05,954 DEBUG   HandlerThread:61045 [meta.py:_save_conda():79] save conda
2022-02-01 10:30:06,494 DEBUG   HandlerThread:61045 [meta.py:_save_conda():89] save conda done
2022-02-01 10:30:06,495 DEBUG   HandlerThread:61045 [meta.py:probe():252] probe done
2022-02-01 10:30:06,500 DEBUG   SenderThread:61045 [sender.py:send():234] send: files
2022-02-01 10:30:06,501 INFO    SenderThread:61045 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 10:30:06,511 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:30:06,512 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:30:06,673 DEBUG   SenderThread:61045 [sender.py:send():234] send: config
2022-02-01 10:30:06,675 DEBUG   SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,676 DEBUG   SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,676 WARNING SenderThread:61045 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 10:30:06,676 DEBUG   SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,676 DEBUG   SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,676 DEBUG   SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,677 DEBUG   SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,677 DEBUG   SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,677 DEBUG   SenderThread:61045 [sender.py:send():234] send: history
2022-02-01 10:30:06,677 DEBUG   SenderThread:61045 [sender.py:send():234] send: summary
2022-02-01 10:30:06,678 INFO    SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 10:30:06,898 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/conda-environment.yaml
2022-02-01 10:30:06,898 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:30:06,898 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json
2022-02-01 10:30:06,898 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-metadata.json
2022-02-01 10:30:06,899 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/requirements.txt
2022-02-01 10:30:07,139 INFO    Thread-11 :61045 [upload_job.py:push():137] Uploaded file /tmp/tmpzeoerhazwandb/25qe2qjl-wandb-metadata.json
2022-02-01 10:30:08,898 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:30:10,899 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:30:21,675 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:30:21,676 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:30:34,180 DEBUG   SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:30:36,839 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:30:36,839 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:30:36,921 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/config.yaml
2022-02-01 10:30:52,000 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:30:52,001 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:04,514 DEBUG   SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:31:07,162 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:31:07,163 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:22,319 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:31:22,320 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:24,949 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:30,952 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:32,953 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:34,785 DEBUG   SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:31:34,954 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:36,955 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:37,488 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:31:37,489 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:38,957 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:52,642 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:31:52,643 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:52,963 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:32:05,018 DEBUG   SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:32:07,832 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:32:07,833 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:32:23,003 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:32:23,003 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:32:35,259 DEBUG   SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:32:38,164 DEBUG   HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:32:38,165 DEBUG   SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:32:42,989 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:32:44,990 INFO    Thread-8  :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log