AlexN's picture
End of training
4641f67
raw
history blame
No virus
6.75 kB
2022-02-01 14:24:48,994 INFO MainThread:29947 [internal.py:wandb_internal():87] W&B internal server running at pid: 29947, started at: 2022-02-01 14:24:48.994283
2022-02-01 14:24:48,998 INFO WriterThread:29947 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/run-3hqii6h6.wandb
2022-02-01 14:24:48,999 DEBUG HandlerThread:29947 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 14:24:49,002 DEBUG SenderThread:29947 [sender.py:send():234] send: header
2022-02-01 14:24:49,002 DEBUG SenderThread:29947 [sender.py:send_request():248] send_request: check_version
2022-02-01 14:24:49,081 DEBUG SenderThread:29947 [sender.py:send():234] send: run
2022-02-01 14:24:49,273 INFO SenderThread:29947 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files
2022-02-01 14:24:49,273 INFO SenderThread:29947 [sender.py:_start_run_threads():804] run started: 3hqii6h6 with start time 1643725488
2022-02-01 14:24:49,273 DEBUG SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:24:49,274 INFO SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:24:49,276 DEBUG HandlerThread:29947 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 14:24:49,335 DEBUG HandlerThread:29947 [meta.py:__init__():40] meta init
2022-02-01 14:24:49,335 DEBUG HandlerThread:29947 [meta.py:__init__():54] meta init done
2022-02-01 14:24:49,335 DEBUG HandlerThread:29947 [meta.py:probe():214] probe
2022-02-01 14:24:49,342 DEBUG HandlerThread:29947 [meta.py:_setup_git():204] setup git
2022-02-01 14:24:49,374 DEBUG HandlerThread:29947 [meta.py:_setup_git():211] setup git done
2022-02-01 14:24:49,374 DEBUG HandlerThread:29947 [meta.py:_save_pip():58] save pip
2022-02-01 14:24:49,375 DEBUG HandlerThread:29947 [meta.py:_save_pip():72] save pip done
2022-02-01 14:24:49,375 DEBUG HandlerThread:29947 [meta.py:_save_conda():79] save conda
2022-02-01 14:24:49,989 DEBUG HandlerThread:29947 [meta.py:_save_conda():89] save conda done
2022-02-01 14:24:49,989 DEBUG HandlerThread:29947 [meta.py:probe():252] probe done
2022-02-01 14:24:49,998 DEBUG SenderThread:29947 [sender.py:send():234] send: files
2022-02-01 14:24:49,999 INFO SenderThread:29947 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 14:24:50,010 DEBUG HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:24:50,011 DEBUG SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:24:50,173 DEBUG SenderThread:29947 [sender.py:send():234] send: config
2022-02-01 14:24:50,176 DEBUG SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,176 DEBUG SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 WARNING SenderThread:29947 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 14:24:50,177 DEBUG SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,178 DEBUG SenderThread:29947 [sender.py:send():234] send: history
2022-02-01 14:24:50,178 DEBUG SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:24:50,178 INFO SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:24:50,276 INFO Thread-8 :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:24:50,277 INFO Thread-8 :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/requirements.txt
2022-02-01 14:24:50,277 INFO Thread-8 :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/conda-environment.yaml
2022-02-01 14:24:50,278 INFO Thread-8 :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-metadata.json
2022-02-01 14:24:50,278 INFO Thread-8 :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json
2022-02-01 14:24:50,617 INFO Thread-11 :29947 [upload_job.py:push():137] Uploaded file /tmp/tmpp56x1syiwandb/1qh281ys-wandb-metadata.json
2022-02-01 14:24:52,275 INFO Thread-8 :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:24:54,276 INFO Thread-8 :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:25:05,175 DEBUG HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:05,176 DEBUG SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:17,592 DEBUG SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:25:20,294 INFO Thread-8 :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml
2022-02-01 14:25:20,335 DEBUG HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:20,335 DEBUG SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:35,486 DEBUG HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:35,487 DEBUG SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:47,927 DEBUG SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:25:50,662 DEBUG HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:50,663 DEBUG SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:26:05,830 DEBUG HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:26:05,830 DEBUG SenderThread:29947 [sender.py:send_request():248] send_request: stop_status