2022-02-01 14:24:48,994 INFO    MainThread:29947 [internal.py:wandb_internal():87] W&B internal server running at pid: 29947, started at: 2022-02-01 14:24:48.994283
2022-02-01 14:24:48,998 INFO    WriterThread:29947 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/run-3hqii6h6.wandb
2022-02-01 14:24:48,999 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 14:24:49,002 DEBUG   SenderThread:29947 [sender.py:send():234] send: header
2022-02-01 14:24:49,002 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: check_version
2022-02-01 14:24:49,081 DEBUG   SenderThread:29947 [sender.py:send():234] send: run
2022-02-01 14:24:49,273 INFO    SenderThread:29947 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files
2022-02-01 14:24:49,273 INFO    SenderThread:29947 [sender.py:_start_run_threads():804] run started: 3hqii6h6 with start time 1643725488
2022-02-01 14:24:49,273 DEBUG   SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:24:49,274 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:24:49,276 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 14:24:49,335 DEBUG   HandlerThread:29947 [meta.py:__init__():40] meta init
2022-02-01 14:24:49,335 DEBUG   HandlerThread:29947 [meta.py:__init__():54] meta init done
2022-02-01 14:24:49,335 DEBUG   HandlerThread:29947 [meta.py:probe():214] probe
2022-02-01 14:24:49,342 DEBUG   HandlerThread:29947 [meta.py:_setup_git():204] setup git
2022-02-01 14:24:49,374 DEBUG   HandlerThread:29947 [meta.py:_setup_git():211] setup git done
2022-02-01 14:24:49,374 DEBUG   HandlerThread:29947 [meta.py:_save_pip():58] save pip
2022-02-01 14:24:49,375 DEBUG   HandlerThread:29947 [meta.py:_save_pip():72] save pip done
2022-02-01 14:24:49,375 DEBUG   HandlerThread:29947 [meta.py:_save_conda():79] save conda
2022-02-01 14:24:49,989 DEBUG   HandlerThread:29947 [meta.py:_save_conda():89] save conda done
2022-02-01 14:24:49,989 DEBUG   HandlerThread:29947 [meta.py:probe():252] probe done
2022-02-01 14:24:49,998 DEBUG   SenderThread:29947 [sender.py:send():234] send: files
2022-02-01 14:24:49,999 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 14:24:50,010 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:24:50,011 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:24:50,173 DEBUG   SenderThread:29947 [sender.py:send():234] send: config
2022-02-01 14:24:50,176 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,176 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 WARNING SenderThread:29947 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,178 DEBUG   SenderThread:29947 [sender.py:send():234] send: history
2022-02-01 14:24:50,178 DEBUG   SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:24:50,178 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:24:50,276 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:24:50,277 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/requirements.txt
2022-02-01 14:24:50,277 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/conda-environment.yaml
2022-02-01 14:24:50,278 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-metadata.json
2022-02-01 14:24:50,278 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json
2022-02-01 14:24:50,617 INFO    Thread-11 :29947 [upload_job.py:push():137] Uploaded file /tmp/tmpp56x1syiwandb/1qh281ys-wandb-metadata.json
2022-02-01 14:24:52,275 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:24:54,276 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:25:05,175 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:05,176 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:17,592 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:25:20,294 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml
2022-02-01 14:25:20,335 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:20,335 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:35,486 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:35,487 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:47,927 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:25:50,662 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:50,663 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:26:05,830 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:26:05,830 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:26:12,326 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:16,328 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:18,207 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:26:18,330 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:23,406 DEBUG   SenderThread:29947 [sender.py:send():234] send: telemetry
2022-02-01 14:26:23,407 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:23,407 DEBUG   SenderThread:29947 [sender.py:send():234] send: exit
2022-02-01 14:26:23,407 INFO    SenderThread:29947 [sender.py:send_exit():366] handling exit code: 0
2022-02-01 14:26:23,407 INFO    SenderThread:29947 [sender.py:send_exit():368] handling runtime: 94
2022-02-01 14:26:23,411 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:26:23,411 INFO    SenderThread:29947 [sender.py:send_exit():374] send defer
2022-02-01 14:26:23,412 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:23,412 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,413 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 0
2022-02-01 14:26:23,413 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,413 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 0
2022-02-01 14:26:23,413 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 1
2022-02-01 14:26:23,414 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,414 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 1
2022-02-01 14:26:23,486 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,486 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 1
2022-02-01 14:26:23,486 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 2
2022-02-01 14:26:23,486 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:26:23,487 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,487 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 2
2022-02-01 14:26:23,488 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,488 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 2
2022-02-01 14:26:23,488 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 3
2022-02-01 14:26:23,488 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,488 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 3
2022-02-01 14:26:23,489 DEBUG   SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:26:23,490 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:26:23,490 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,490 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 3
2022-02-01 14:26:23,490 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 4
2022-02-01 14:26:23,491 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,491 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 4
2022-02-01 14:26:23,491 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,491 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 4
2022-02-01 14:26:23,515 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:23,724 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 5
2022-02-01 14:26:23,724 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:23,726 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,726 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 5
2022-02-01 14:26:23,726 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,726 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 5
2022-02-01 14:26:23,726 INFO    SenderThread:29947 [dir_watcher.py:finish():283] shutting down directory watcher
2022-02-01 14:26:23,827 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,332 INFO    SenderThread:29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:24,333 INFO    SenderThread:29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json
2022-02-01 14:26:24,334 INFO    SenderThread:29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml
2022-02-01 14:26:24,334 INFO    SenderThread:29947 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files
2022-02-01 14:26:24,334 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log output.log
2022-02-01 14:26:24,335 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-metadata.json wandb-metadata.json
2022-02-01 14:26:24,335 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml config.yaml
2022-02-01 14:26:24,341 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/conda-environment.yaml conda-environment.yaml
2022-02-01 14:26:24,348 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json wandb-summary.json
2022-02-01 14:26:24,348 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/requirements.txt requirements.txt
2022-02-01 14:26:24,349 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 6
2022-02-01 14:26:24,350 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,357 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:24,357 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 6
2022-02-01 14:26:24,357 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:24,358 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 6
2022-02-01 14:26:24,358 INFO    SenderThread:29947 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 14:26:24,452 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,453 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,555 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,560 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,667 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,668 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,770 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,771 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,873 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,874 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,888 INFO    Thread-12 :29947 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:24,897 INFO    Thread-13 :29947 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml
2022-02-01 14:26:24,913 INFO    Thread-15 :29947 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/requirements.txt
2022-02-01 14:26:24,919 INFO    Thread-14 :29947 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json
2022-02-01 14:26:24,976 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,976 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:25,079 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:25,079 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:25,120 INFO    Thread-7  :29947 [sender.py:transition_state():387] send defer: 7
2022-02-01 14:26:25,122 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:25,122 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 7
2022-02-01 14:26:25,122 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:25,122 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 7
2022-02-01 14:26:25,182 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:25,552 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 8
2022-02-01 14:26:25,553 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:25,554 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:25,554 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 8
2022-02-01 14:26:25,555 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:25,555 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 8
2022-02-01 14:26:25,555 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 9
2022-02-01 14:26:25,556 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:25,557 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 9
2022-02-01 14:26:25,557 DEBUG   SenderThread:29947 [sender.py:send():234] send: final
2022-02-01 14:26:25,557 DEBUG   SenderThread:29947 [sender.py:send():234] send: footer
2022-02-01 14:26:25,557 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:25,557 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 9
2022-02-01 14:26:25,656 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:25,656 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:25,657 INFO    SenderThread:29947 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 14:26:25,966 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: get_summary
2022-02-01 14:26:25,968 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: sampled_history
2022-02-01 14:26:25,970 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: shutdown
2022-02-01 14:26:25,970 INFO    HandlerThread:29947 [handler.py:finish():731] shutting down handler
2022-02-01 14:26:26,557 INFO    WriterThread:29947 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/run-3hqii6h6.wandb
2022-02-01 14:26:26,963 INFO    SenderThread:29947 [sender.py:finish():1070] shutting down sender
2022-02-01 14:26:26,964 INFO    SenderThread:29947 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 14:26:26,964 INFO    SenderThread:29947 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 14:26:26,967 INFO    MainThread:29947 [internal.py:handle_exit():77] Internal process exited