File size: 24,414 Bytes
ef5fb31 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 |
2022-01-30 17:51:36,839 INFO MainThread:21293 [internal.py:wandb_internal():87] W&B internal server running at pid: 21293, started at: 2022-01-30 17:51:36.839410
2022-01-30 17:51:36,843 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: check_version
2022-01-30 17:51:36,844 INFO WriterThread:21293 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/run-mz3dbiz4.wandb
2022-01-30 17:51:36,847 DEBUG SenderThread:21293 [sender.py:send():234] send: header
2022-01-30 17:51:36,848 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: check_version
2022-01-30 17:51:36,917 DEBUG SenderThread:21293 [sender.py:send():234] send: run
2022-01-30 17:51:37,133 INFO SenderThread:21293 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files
2022-01-30 17:51:37,134 INFO SenderThread:21293 [sender.py:_start_run_threads():804] run started: mz3dbiz4 with start time 1643565096
2022-01-30 17:51:37,134 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: run_start
2022-01-30 17:51:37,134 DEBUG SenderThread:21293 [sender.py:send():234] send: summary
2022-01-30 17:51:37,136 INFO SenderThread:21293 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:51:37,144 DEBUG HandlerThread:21293 [meta.py:__init__():40] meta init
2022-01-30 17:51:37,144 DEBUG HandlerThread:21293 [meta.py:__init__():54] meta init done
2022-01-30 17:51:37,144 DEBUG HandlerThread:21293 [meta.py:probe():214] probe
2022-01-30 17:51:37,154 DEBUG HandlerThread:21293 [meta.py:_setup_git():204] setup git
2022-01-30 17:51:37,190 DEBUG HandlerThread:21293 [meta.py:_setup_git():211] setup git done
2022-01-30 17:51:37,191 DEBUG HandlerThread:21293 [meta.py:_save_pip():58] save pip
2022-01-30 17:51:37,193 DEBUG HandlerThread:21293 [meta.py:_save_pip():72] save pip done
2022-01-30 17:51:37,193 DEBUG HandlerThread:21293 [meta.py:_save_conda():79] save conda
2022-01-30 17:51:37,717 DEBUG HandlerThread:21293 [meta.py:_save_conda():89] save conda done
2022-01-30 17:51:37,718 DEBUG HandlerThread:21293 [meta.py:probe():252] probe done
2022-01-30 17:51:37,726 DEBUG SenderThread:21293 [sender.py:send():234] send: files
2022-01-30 17:51:37,726 INFO SenderThread:21293 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-30 17:51:37,736 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:51:37,737 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:51:37,890 DEBUG SenderThread:21293 [sender.py:send():234] send: config
2022-01-30 17:51:37,892 DEBUG SenderThread:21293 [sender.py:send():234] send: metric
2022-01-30 17:51:37,893 DEBUG SenderThread:21293 [sender.py:send():234] send: metric
2022-01-30 17:51:37,893 WARNING SenderThread:21293 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-01-30 17:51:38,136 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/conda-environment.yaml
2022-01-30 17:51:38,137 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-summary.json
2022-01-30 17:51:38,137 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/requirements.txt
2022-01-30 17:51:38,138 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:51:38,138 INFO Thread-8 :21293 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-metadata.json
2022-01-30 17:51:38,250 INFO Thread-11 :21293 [upload_job.py:push():137] Uploaded file /tmp/tmp7m8cqty2wandb/246jybel-wandb-metadata.json
2022-01-30 17:51:40,135 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:51:46,138 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:51:48,140 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:51:50,142 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:51:53,043 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:51:53,043 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:51:56,145 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:00,148 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:02,149 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:05,410 DEBUG SenderThread:21293 [sender.py:send():234] send: stats
2022-01-30 17:52:06,151 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:08,153 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/config.yaml
2022-01-30 17:52:08,539 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:52:08,540 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:52:10,154 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:12,156 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:16,158 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:21,162 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:23,163 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:23,731 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:52:23,732 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:52:29,168 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:31,169 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:33,170 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:35,634 DEBUG SenderThread:21293 [sender.py:send():234] send: stats
2022-01-30 17:52:37,174 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:38,896 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:52:38,896 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:52:43,178 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:45,179 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:47,180 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:53,183 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:54,057 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:52:54,058 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:52:57,186 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:52:59,187 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:03,190 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:05,866 DEBUG SenderThread:21293 [sender.py:send():234] send: stats
2022-01-30 17:53:07,192 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:09,194 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:09,222 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:53:09,223 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:53:15,197 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:17,198 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:19,200 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:23,992 WARNING MainThread:21293 [internal.py:wandb_internal():146] Internal process interrupt: 1
2022-01-30 17:53:25,205 INFO Thread-8 :21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:26,380 DEBUG SenderThread:21293 [sender.py:send():234] send: telemetry
2022-01-30 17:53:26,380 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:26,380 DEBUG SenderThread:21293 [sender.py:send():234] send: exit
2022-01-30 17:53:26,381 INFO SenderThread:21293 [sender.py:send_exit():366] handling exit code: 255
2022-01-30 17:53:26,381 INFO SenderThread:21293 [sender.py:send_exit():368] handling runtime: 109
2022-01-30 17:53:26,381 INFO SenderThread:21293 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:53:26,382 INFO SenderThread:21293 [sender.py:send_exit():374] send defer
2022-01-30 17:53:26,382 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:26,383 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:26,383 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-30 17:53:26,384 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:26,384 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-30 17:53:26,384 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 1
2022-01-30 17:53:26,384 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:26,385 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-30 17:53:26,424 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:26,424 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-30 17:53:26,424 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 2
2022-01-30 17:53:26,425 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:26,425 DEBUG SenderThread:21293 [sender.py:send():234] send: stats
2022-01-30 17:53:26,426 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-30 17:53:26,427 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:26,427 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-30 17:53:26,427 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 3
2022-01-30 17:53:26,427 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:26,428 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-30 17:53:26,428 DEBUG SenderThread:21293 [sender.py:send():234] send: summary
2022-01-30 17:53:26,429 INFO SenderThread:21293 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:53:26,429 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:26,429 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-30 17:53:26,430 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 4
2022-01-30 17:53:26,430 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:26,430 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-30 17:53:26,431 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:26,431 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-30 17:53:26,485 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:26,670 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 5
2022-01-30 17:53:26,670 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:26,670 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:26,671 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-30 17:53:26,671 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:26,671 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-30 17:53:26,671 INFO SenderThread:21293 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-30 17:53:26,772 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:27,207 INFO SenderThread:21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/config.yaml
2022-01-30 17:53:27,207 INFO SenderThread:21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:27,208 INFO SenderThread:21293 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-summary.json
2022-01-30 17:53:27,208 INFO SenderThread:21293 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files
2022-01-30 17:53:27,209 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log output.log
2022-01-30 17:53:27,209 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-metadata.json wandb-metadata.json
2022-01-30 17:53:27,209 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/config.yaml config.yaml
2022-01-30 17:53:27,210 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/conda-environment.yaml conda-environment.yaml
2022-01-30 17:53:27,222 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-summary.json wandb-summary.json
2022-01-30 17:53:27,223 INFO SenderThread:21293 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/requirements.txt requirements.txt
2022-01-30 17:53:27,224 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 6
2022-01-30 17:53:27,224 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:27,233 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:27,233 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-30 17:53:27,241 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:27,242 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-30 17:53:27,242 INFO SenderThread:21293 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 17:53:27,329 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:27,329 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:27,436 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:27,440 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:27,543 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:27,543 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:27,646 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:27,646 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:27,738 INFO Thread-12 :21293 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/output.log
2022-01-30 17:53:27,746 INFO Thread-14 :21293 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/wandb-summary.json
2022-01-30 17:53:27,748 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:27,748 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:27,761 INFO Thread-13 :21293 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/config.yaml
2022-01-30 17:53:27,775 INFO Thread-15 :21293 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/files/requirements.txt
2022-01-30 17:53:27,851 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:27,851 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:27,954 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:27,954 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:27,976 INFO Thread-7 :21293 [sender.py:transition_state():387] send defer: 7
2022-01-30 17:53:27,977 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:27,977 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-30 17:53:27,977 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:27,977 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-30 17:53:28,057 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:28,973 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 8
2022-01-30 17:53:28,974 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:28,974 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:28,975 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-30 17:53:28,975 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:28,975 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-30 17:53:28,975 INFO SenderThread:21293 [sender.py:transition_state():387] send defer: 9
2022-01-30 17:53:28,976 DEBUG SenderThread:21293 [sender.py:send():234] send: final
2022-01-30 17:53:28,977 DEBUG SenderThread:21293 [sender.py:send():234] send: footer
2022-01-30 17:53:28,977 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:53:28,977 INFO HandlerThread:21293 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-30 17:53:28,977 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: defer
2022-01-30 17:53:28,977 INFO SenderThread:21293 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-30 17:53:29,077 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:53:29,077 DEBUG SenderThread:21293 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:53:29,078 INFO SenderThread:21293 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 17:53:29,426 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: get_summary
2022-01-30 17:53:29,427 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-30 17:53:29,429 DEBUG HandlerThread:21293 [handler.py:handle_request():130] handle_request: shutdown
2022-01-30 17:53:29,429 INFO HandlerThread:21293 [handler.py:finish():731] shutting down handler
2022-01-30 17:53:29,977 INFO WriterThread:21293 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_175135-mz3dbiz4/run-mz3dbiz4.wandb
2022-01-30 17:53:30,424 INFO SenderThread:21293 [sender.py:finish():1070] shutting down sender
2022-01-30 17:53:30,425 INFO SenderThread:21293 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 17:53:30,425 INFO SenderThread:21293 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 17:53:30,428 INFO MainThread:21293 [internal.py:handle_exit():77] Internal process exited
|