2022-02-01 14:24:48,994 INFO    MainThread:29947 [internal.py:wandb_internal():87] W&B internal server running at pid: 29947, started at: 2022-02-01 14:24:48.994283
2022-02-01 14:24:48,998 INFO    WriterThread:29947 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/run-3hqii6h6.wandb
2022-02-01 14:24:48,999 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 14:24:49,002 DEBUG   SenderThread:29947 [sender.py:send():234] send: header
2022-02-01 14:24:49,002 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: check_version
2022-02-01 14:24:49,081 DEBUG   SenderThread:29947 [sender.py:send():234] send: run
2022-02-01 14:24:49,273 INFO    SenderThread:29947 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files
2022-02-01 14:24:49,273 INFO    SenderThread:29947 [sender.py:_start_run_threads():804] run started: 3hqii6h6 with start time 1643725488
2022-02-01 14:24:49,273 DEBUG   SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:24:49,274 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:24:49,276 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 14:24:49,335 DEBUG   HandlerThread:29947 [meta.py:__init__():40] meta init
2022-02-01 14:24:49,335 DEBUG   HandlerThread:29947 [meta.py:__init__():54] meta init done
2022-02-01 14:24:49,335 DEBUG   HandlerThread:29947 [meta.py:probe():214] probe
2022-02-01 14:24:49,342 DEBUG   HandlerThread:29947 [meta.py:_setup_git():204] setup git
2022-02-01 14:24:49,374 DEBUG   HandlerThread:29947 [meta.py:_setup_git():211] setup git done
2022-02-01 14:24:49,374 DEBUG   HandlerThread:29947 [meta.py:_save_pip():58] save pip
2022-02-01 14:24:49,375 DEBUG   HandlerThread:29947 [meta.py:_save_pip():72] save pip done
2022-02-01 14:24:49,375 DEBUG   HandlerThread:29947 [meta.py:_save_conda():79] save conda
2022-02-01 14:24:49,989 DEBUG   HandlerThread:29947 [meta.py:_save_conda():89] save conda done
2022-02-01 14:24:49,989 DEBUG   HandlerThread:29947 [meta.py:probe():252] probe done
2022-02-01 14:24:49,998 DEBUG   SenderThread:29947 [sender.py:send():234] send: files
2022-02-01 14:24:49,999 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 14:24:50,010 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:24:50,011 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:24:50,173 DEBUG   SenderThread:29947 [sender.py:send():234] send: config
2022-02-01 14:24:50,176 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,176 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 WARNING SenderThread:29947 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,178 DEBUG   SenderThread:29947 [sender.py:send():234] send: history
2022-02-01 14:24:50,178 DEBUG   SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:24:50,178 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:24:50,276 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:24:50,277 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/requirements.txt
2022-02-01 14:24:50,277 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/conda-environment.yaml
2022-02-01 14:24:50,278 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-metadata.json
2022-02-01 14:24:50,278 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json
2022-02-01 14:24:50,617 INFO    Thread-11 :29947 [upload_job.py:push():137] Uploaded file /tmp/tmpp56x1syiwandb/1qh281ys-wandb-metadata.json
2022-02-01 14:24:52,275 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:24:54,276 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:25:05,175 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:05,176 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:17,592 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:25:20,294 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml
2022-02-01 14:25:20,335 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:20,335 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:35,486 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:35,487 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:47,927 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:25:50,662 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:50,663 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:26:05,830 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:26:05,830 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status