| 2022-01-30 17:47:50,131 INFO MainThread:19599 [internal.py:wandb_internal():87] W&B internal server running at pid: 19599, started at: 2022-01-30 17:47:50.130774 | |
| 2022-01-30 17:47:50,135 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: check_version | |
| 2022-01-30 17:47:50,135 INFO WriterThread:19599 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/run-kzuygp3o.wandb | |
| 2022-01-30 17:47:50,139 DEBUG SenderThread:19599 [sender.py:send():234] send: header | |
| 2022-01-30 17:47:50,139 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: check_version | |
| 2022-01-30 17:47:50,215 DEBUG SenderThread:19599 [sender.py:send():234] send: run | |
| 2022-01-30 17:47:50,957 INFO SenderThread:19599 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files | |
| 2022-01-30 17:47:50,957 INFO SenderThread:19599 [sender.py:_start_run_threads():804] run started: kzuygp3o with start time 1643564869 | |
| 2022-01-30 17:47:50,958 DEBUG SenderThread:19599 [sender.py:send():234] send: summary | |
| 2022-01-30 17:47:50,958 INFO SenderThread:19599 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-30 17:47:50,959 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: run_start | |
| 2022-01-30 17:47:50,967 DEBUG HandlerThread:19599 [meta.py:__init__():40] meta init | |
| 2022-01-30 17:47:50,967 DEBUG HandlerThread:19599 [meta.py:__init__():54] meta init done | |
| 2022-01-30 17:47:50,968 DEBUG HandlerThread:19599 [meta.py:probe():214] probe | |
| 2022-01-30 17:47:50,976 DEBUG HandlerThread:19599 [meta.py:_setup_git():204] setup git | |
| 2022-01-30 17:47:51,010 DEBUG HandlerThread:19599 [meta.py:_setup_git():211] setup git done | |
| 2022-01-30 17:47:51,010 DEBUG HandlerThread:19599 [meta.py:_save_pip():58] save pip | |
| 2022-01-30 17:47:51,011 DEBUG HandlerThread:19599 [meta.py:_save_pip():72] save pip done | |
| 2022-01-30 17:47:51,011 DEBUG HandlerThread:19599 [meta.py:_save_conda():79] save conda | |
| 2022-01-30 17:47:51,503 DEBUG HandlerThread:19599 [meta.py:_save_conda():89] save conda done | |
| 2022-01-30 17:47:51,503 DEBUG HandlerThread:19599 [meta.py:probe():252] probe done | |
| 2022-01-30 17:47:51,511 DEBUG SenderThread:19599 [sender.py:send():234] send: files | |
| 2022-01-30 17:47:51,511 INFO SenderThread:19599 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now | |
| 2022-01-30 17:47:51,523 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-30 17:47:51,524 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-30 17:47:51,704 DEBUG SenderThread:19599 [sender.py:send():234] send: config | |
| 2022-01-30 17:47:51,706 DEBUG SenderThread:19599 [sender.py:send():234] send: metric | |
| 2022-01-30 17:47:51,707 DEBUG SenderThread:19599 [sender.py:send():234] send: metric | |
| 2022-01-30 17:47:51,707 WARNING SenderThread:19599 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen) | |
| 2022-01-30 17:47:51,961 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-metadata.json | |
| 2022-01-30 17:47:51,961 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/requirements.txt | |
| 2022-01-30 17:47:51,961 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log | |
| 2022-01-30 17:47:51,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json | |
| 2022-01-30 17:47:51,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/conda-environment.yaml | |
| 2022-01-30 17:47:52,037 INFO Thread-11 :19599 [upload_job.py:push():137] Uploaded file /tmp/tmpl83m3xpuwandb/2em52v7y-wandb-metadata.json | |
| 2022-01-30 17:47:53,959 INFO Thread-8 :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log | |
| 2022-01-30 17:47:57,751 DEBUG SenderThread:19599 [sender.py:send():234] send: telemetry | |
| 2022-01-30 17:47:57,752 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:57,752 DEBUG SenderThread:19599 [sender.py:send():234] send: exit | |
| 2022-01-30 17:47:57,753 INFO SenderThread:19599 [sender.py:send_exit():366] handling exit code: 1 | |
| 2022-01-30 17:47:57,753 INFO SenderThread:19599 [sender.py:send_exit():368] handling runtime: 6 | |
| 2022-01-30 17:47:57,754 INFO SenderThread:19599 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-30 17:47:57,754 INFO SenderThread:19599 [sender.py:send_exit():374] send defer | |
| 2022-01-30 17:47:57,754 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:57,755 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:47:57,755 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 0 | |
| 2022-01-30 17:47:57,756 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:47:57,756 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 0 | |
| 2022-01-30 17:47:57,756 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 1 | |
| 2022-01-30 17:47:57,757 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:47:57,757 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 1 | |
| 2022-01-30 17:47:57,792 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:47:57,793 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 1 | |
| 2022-01-30 17:47:57,793 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 2 | |
| 2022-01-30 17:47:57,794 DEBUG SenderThread:19599 [sender.py:send():234] send: stats | |
| 2022-01-30 17:47:57,795 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:47:57,795 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 2 | |
| 2022-01-30 17:47:57,795 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:47:57,796 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 2 | |
| 2022-01-30 17:47:57,796 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 3 | |
| 2022-01-30 17:47:57,796 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:47:57,797 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 3 | |
| 2022-01-30 17:47:57,797 DEBUG SenderThread:19599 [sender.py:send():234] send: summary | |
| 2022-01-30 17:47:57,798 INFO SenderThread:19599 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-30 17:47:57,798 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:47:57,798 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 3 | |
| 2022-01-30 17:47:57,798 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 4 | |
| 2022-01-30 17:47:57,799 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:47:57,799 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 4 | |
| 2022-01-30 17:47:57,799 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:47:57,799 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 4 | |
| 2022-01-30 17:47:57,857 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:57,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json | |
| 2022-01-30 17:47:57,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log | |
| 2022-01-30 17:47:58,077 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 5 | |
| 2022-01-30 17:47:58,077 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:58,078 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:47:58,079 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 5 | |
| 2022-01-30 17:47:58,079 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:47:58,079 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 5 | |
| 2022-01-30 17:47:58,079 INFO SenderThread:19599 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-01-30 17:47:58,180 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:58,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/config.yaml | |
| 2022-01-30 17:47:58,964 INFO SenderThread:19599 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files | |
| 2022-01-30 17:47:58,965 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log output.log | |
| 2022-01-30 17:47:58,965 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-metadata.json wandb-metadata.json | |
| 2022-01-30 17:47:58,966 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/config.yaml config.yaml | |
| 2022-01-30 17:47:58,970 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/conda-environment.yaml conda-environment.yaml | |
| 2022-01-30 17:47:58,977 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json wandb-summary.json | |
| 2022-01-30 17:47:58,978 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/requirements.txt requirements.txt | |
| 2022-01-30 17:47:58,985 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 6 | |
| 2022-01-30 17:47:58,986 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:58,986 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:47:58,987 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 6 | |
| 2022-01-30 17:47:58,988 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:47:58,993 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 6 | |
| 2022-01-30 17:47:58,994 INFO SenderThread:19599 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-01-30 17:47:59,089 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:59,089 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:59,192 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:59,192 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:59,295 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:59,296 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:59,399 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:59,399 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:59,490 INFO Thread-14 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json | |
| 2022-01-30 17:47:59,495 INFO Thread-15 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/requirements.txt | |
| 2022-01-30 17:47:59,502 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:59,502 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:59,513 INFO Thread-12 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log | |
| 2022-01-30 17:47:59,591 INFO Thread-13 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/config.yaml | |
| 2022-01-30 17:47:59,604 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:59,605 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:59,707 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:47:59,707 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:47:59,793 INFO Thread-7 :19599 [sender.py:transition_state():387] send defer: 7 | |
| 2022-01-30 17:47:59,794 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:47:59,794 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 7 | |
| 2022-01-30 17:47:59,794 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:47:59,794 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 7 | |
| 2022-01-30 17:47:59,809 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:48:00,234 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 8 | |
| 2022-01-30 17:48:00,235 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:48:00,236 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:48:00,236 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 8 | |
| 2022-01-30 17:48:00,237 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:48:00,237 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 8 | |
| 2022-01-30 17:48:00,237 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 9 | |
| 2022-01-30 17:48:00,238 DEBUG SenderThread:19599 [sender.py:send():234] send: final | |
| 2022-01-30 17:48:00,238 DEBUG SenderThread:19599 [sender.py:send():234] send: footer | |
| 2022-01-30 17:48:00,239 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 17:48:00,239 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 9 | |
| 2022-01-30 17:48:00,239 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 17:48:00,239 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 9 | |
| 2022-01-30 17:48:00,338 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 17:48:00,338 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 17:48:00,338 INFO SenderThread:19599 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-01-30 17:48:00,657 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: get_summary | |
| 2022-01-30 17:48:00,659 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: sampled_history | |
| 2022-01-30 17:48:00,660 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: shutdown | |
| 2022-01-30 17:48:00,660 INFO HandlerThread:19599 [handler.py:finish():731] shutting down handler | |
| 2022-01-30 17:48:01,238 INFO WriterThread:19599 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/run-kzuygp3o.wandb | |
| 2022-01-30 17:48:01,655 INFO SenderThread:19599 [sender.py:finish():1070] shutting down sender | |
| 2022-01-30 17:48:01,655 INFO SenderThread:19599 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-01-30 17:48:01,655 INFO SenderThread:19599 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-01-30 17:48:01,659 INFO MainThread:19599 [internal.py:handle_exit():77] Internal process exited | |