File size: 17,119 Bytes
ef5fb31 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 |
2022-01-30 17:47:50,131 INFO MainThread:19599 [internal.py:wandb_internal():87] W&B internal server running at pid: 19599, started at: 2022-01-30 17:47:50.130774
2022-01-30 17:47:50,135 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: check_version
2022-01-30 17:47:50,135 INFO WriterThread:19599 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/run-kzuygp3o.wandb
2022-01-30 17:47:50,139 DEBUG SenderThread:19599 [sender.py:send():234] send: header
2022-01-30 17:47:50,139 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: check_version
2022-01-30 17:47:50,215 DEBUG SenderThread:19599 [sender.py:send():234] send: run
2022-01-30 17:47:50,957 INFO SenderThread:19599 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files
2022-01-30 17:47:50,957 INFO SenderThread:19599 [sender.py:_start_run_threads():804] run started: kzuygp3o with start time 1643564869
2022-01-30 17:47:50,958 DEBUG SenderThread:19599 [sender.py:send():234] send: summary
2022-01-30 17:47:50,958 INFO SenderThread:19599 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:47:50,959 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: run_start
2022-01-30 17:47:50,967 DEBUG HandlerThread:19599 [meta.py:__init__():40] meta init
2022-01-30 17:47:50,967 DEBUG HandlerThread:19599 [meta.py:__init__():54] meta init done
2022-01-30 17:47:50,968 DEBUG HandlerThread:19599 [meta.py:probe():214] probe
2022-01-30 17:47:50,976 DEBUG HandlerThread:19599 [meta.py:_setup_git():204] setup git
2022-01-30 17:47:51,010 DEBUG HandlerThread:19599 [meta.py:_setup_git():211] setup git done
2022-01-30 17:47:51,010 DEBUG HandlerThread:19599 [meta.py:_save_pip():58] save pip
2022-01-30 17:47:51,011 DEBUG HandlerThread:19599 [meta.py:_save_pip():72] save pip done
2022-01-30 17:47:51,011 DEBUG HandlerThread:19599 [meta.py:_save_conda():79] save conda
2022-01-30 17:47:51,503 DEBUG HandlerThread:19599 [meta.py:_save_conda():89] save conda done
2022-01-30 17:47:51,503 DEBUG HandlerThread:19599 [meta.py:probe():252] probe done
2022-01-30 17:47:51,511 DEBUG SenderThread:19599 [sender.py:send():234] send: files
2022-01-30 17:47:51,511 INFO SenderThread:19599 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-30 17:47:51,523 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:47:51,524 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:47:51,704 DEBUG SenderThread:19599 [sender.py:send():234] send: config
2022-01-30 17:47:51,706 DEBUG SenderThread:19599 [sender.py:send():234] send: metric
2022-01-30 17:47:51,707 DEBUG SenderThread:19599 [sender.py:send():234] send: metric
2022-01-30 17:47:51,707 WARNING SenderThread:19599 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-01-30 17:47:51,961 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-metadata.json
2022-01-30 17:47:51,961 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/requirements.txt
2022-01-30 17:47:51,961 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log
2022-01-30 17:47:51,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json
2022-01-30 17:47:51,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/conda-environment.yaml
2022-01-30 17:47:52,037 INFO Thread-11 :19599 [upload_job.py:push():137] Uploaded file /tmp/tmpl83m3xpuwandb/2em52v7y-wandb-metadata.json
2022-01-30 17:47:53,959 INFO Thread-8 :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log
2022-01-30 17:47:57,751 DEBUG SenderThread:19599 [sender.py:send():234] send: telemetry
2022-01-30 17:47:57,752 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:57,752 DEBUG SenderThread:19599 [sender.py:send():234] send: exit
2022-01-30 17:47:57,753 INFO SenderThread:19599 [sender.py:send_exit():366] handling exit code: 1
2022-01-30 17:47:57,753 INFO SenderThread:19599 [sender.py:send_exit():368] handling runtime: 6
2022-01-30 17:47:57,754 INFO SenderThread:19599 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:47:57,754 INFO SenderThread:19599 [sender.py:send_exit():374] send defer
2022-01-30 17:47:57,754 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:57,755 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,755 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-30 17:47:57,756 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,756 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-30 17:47:57,756 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 1
2022-01-30 17:47:57,757 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,757 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-30 17:47:57,792 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,793 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-30 17:47:57,793 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 2
2022-01-30 17:47:57,794 DEBUG SenderThread:19599 [sender.py:send():234] send: stats
2022-01-30 17:47:57,795 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,795 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-30 17:47:57,795 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,796 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-30 17:47:57,796 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 3
2022-01-30 17:47:57,796 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,797 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-30 17:47:57,797 DEBUG SenderThread:19599 [sender.py:send():234] send: summary
2022-01-30 17:47:57,798 INFO SenderThread:19599 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:47:57,798 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,798 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-30 17:47:57,798 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 4
2022-01-30 17:47:57,799 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,799 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-30 17:47:57,799 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,799 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-30 17:47:57,857 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:57,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json
2022-01-30 17:47:57,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log
2022-01-30 17:47:58,077 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 5
2022-01-30 17:47:58,077 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:58,078 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:58,079 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-30 17:47:58,079 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:58,079 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-30 17:47:58,079 INFO SenderThread:19599 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-30 17:47:58,180 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:58,962 INFO Thread-8 :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/config.yaml
2022-01-30 17:47:58,964 INFO SenderThread:19599 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files
2022-01-30 17:47:58,965 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log output.log
2022-01-30 17:47:58,965 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-metadata.json wandb-metadata.json
2022-01-30 17:47:58,966 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/config.yaml config.yaml
2022-01-30 17:47:58,970 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/conda-environment.yaml conda-environment.yaml
2022-01-30 17:47:58,977 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json wandb-summary.json
2022-01-30 17:47:58,978 INFO SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/requirements.txt requirements.txt
2022-01-30 17:47:58,985 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 6
2022-01-30 17:47:58,986 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:58,986 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:58,987 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-30 17:47:58,988 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:58,993 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-30 17:47:58,994 INFO SenderThread:19599 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 17:47:59,089 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,089 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,192 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,192 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,295 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,296 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,399 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,399 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,490 INFO Thread-14 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json
2022-01-30 17:47:59,495 INFO Thread-15 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/requirements.txt
2022-01-30 17:47:59,502 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,502 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,513 INFO Thread-12 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log
2022-01-30 17:47:59,591 INFO Thread-13 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/config.yaml
2022-01-30 17:47:59,604 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,605 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,707 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,707 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,793 INFO Thread-7 :19599 [sender.py:transition_state():387] send defer: 7
2022-01-30 17:47:59,794 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:59,794 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-30 17:47:59,794 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:59,794 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-30 17:47:59,809 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:48:00,234 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 8
2022-01-30 17:48:00,235 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:48:00,236 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:48:00,236 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-30 17:48:00,237 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:48:00,237 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-30 17:48:00,237 INFO SenderThread:19599 [sender.py:transition_state():387] send defer: 9
2022-01-30 17:48:00,238 DEBUG SenderThread:19599 [sender.py:send():234] send: final
2022-01-30 17:48:00,238 DEBUG SenderThread:19599 [sender.py:send():234] send: footer
2022-01-30 17:48:00,239 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:48:00,239 INFO HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-30 17:48:00,239 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:48:00,239 INFO SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-30 17:48:00,338 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:48:00,338 DEBUG SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:48:00,338 INFO SenderThread:19599 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 17:48:00,657 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: get_summary
2022-01-30 17:48:00,659 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-30 17:48:00,660 DEBUG HandlerThread:19599 [handler.py:handle_request():130] handle_request: shutdown
2022-01-30 17:48:00,660 INFO HandlerThread:19599 [handler.py:finish():731] shutting down handler
2022-01-30 17:48:01,238 INFO WriterThread:19599 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/run-kzuygp3o.wandb
2022-01-30 17:48:01,655 INFO SenderThread:19599 [sender.py:finish():1070] shutting down sender
2022-01-30 17:48:01,655 INFO SenderThread:19599 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 17:48:01,655 INFO SenderThread:19599 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 17:48:01,659 INFO MainThread:19599 [internal.py:handle_exit():77] Internal process exited
|