File size: 22,938 Bytes
f25a924 a37375f 4641f67 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 |
2022-02-01 10:30:05,624 INFO MainThread:61045 [internal.py:wandb_internal():87] W&B internal server running at pid: 61045, started at: 2022-02-01 10:30:05.624191
2022-02-01 10:30:05,628 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 10:30:05,628 INFO WriterThread:61045 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/run-1yfj7vwy.wandb
2022-02-01 10:30:05,632 DEBUG SenderThread:61045 [sender.py:send():234] send: header
2022-02-01 10:30:05,632 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: check_version
2022-02-01 10:30:05,706 DEBUG SenderThread:61045 [sender.py:send():234] send: run
2022-02-01 10:30:05,896 INFO SenderThread:61045 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files
2022-02-01 10:30:05,896 INFO SenderThread:61045 [sender.py:_start_run_threads():804] run started: 1yfj7vwy with start time 1643711404
2022-02-01 10:30:05,896 DEBUG SenderThread:61045 [sender.py:send():234] send: summary
2022-02-01 10:30:05,897 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 10:30:05,898 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 10:30:05,908 DEBUG HandlerThread:61045 [meta.py:__init__():40] meta init
2022-02-01 10:30:05,909 DEBUG HandlerThread:61045 [meta.py:__init__():54] meta init done
2022-02-01 10:30:05,909 DEBUG HandlerThread:61045 [meta.py:probe():214] probe
2022-02-01 10:30:05,917 DEBUG HandlerThread:61045 [meta.py:_setup_git():204] setup git
2022-02-01 10:30:05,952 DEBUG HandlerThread:61045 [meta.py:_setup_git():211] setup git done
2022-02-01 10:30:05,953 DEBUG HandlerThread:61045 [meta.py:_save_pip():58] save pip
2022-02-01 10:30:05,954 DEBUG HandlerThread:61045 [meta.py:_save_pip():72] save pip done
2022-02-01 10:30:05,954 DEBUG HandlerThread:61045 [meta.py:_save_conda():79] save conda
2022-02-01 10:30:06,494 DEBUG HandlerThread:61045 [meta.py:_save_conda():89] save conda done
2022-02-01 10:30:06,495 DEBUG HandlerThread:61045 [meta.py:probe():252] probe done
2022-02-01 10:30:06,500 DEBUG SenderThread:61045 [sender.py:send():234] send: files
2022-02-01 10:30:06,501 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 10:30:06,511 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:30:06,512 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:30:06,673 DEBUG SenderThread:61045 [sender.py:send():234] send: config
2022-02-01 10:30:06,675 DEBUG SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,676 DEBUG SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,676 WARNING SenderThread:61045 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 10:30:06,676 DEBUG SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,676 DEBUG SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,676 DEBUG SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,677 DEBUG SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,677 DEBUG SenderThread:61045 [sender.py:send():234] send: metric
2022-02-01 10:30:06,677 DEBUG SenderThread:61045 [sender.py:send():234] send: history
2022-02-01 10:30:06,677 DEBUG SenderThread:61045 [sender.py:send():234] send: summary
2022-02-01 10:30:06,678 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 10:30:06,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/conda-environment.yaml
2022-02-01 10:30:06,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:30:06,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json
2022-02-01 10:30:06,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-metadata.json
2022-02-01 10:30:06,899 INFO Thread-8 :61045 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/requirements.txt
2022-02-01 10:30:07,139 INFO Thread-11 :61045 [upload_job.py:push():137] Uploaded file /tmp/tmpzeoerhazwandb/25qe2qjl-wandb-metadata.json
2022-02-01 10:30:08,898 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:30:10,899 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:30:21,675 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:30:21,676 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:30:34,180 DEBUG SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:30:36,839 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:30:36,839 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:30:36,921 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/config.yaml
2022-02-01 10:30:52,000 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:30:52,001 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:04,514 DEBUG SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:31:07,162 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:31:07,163 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:22,319 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:31:22,320 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:24,949 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:30,952 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:32,953 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:34,785 DEBUG SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:31:34,954 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:36,955 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:37,488 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:31:37,489 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:38,957 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:31:52,642 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:31:52,643 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:31:52,963 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:32:05,018 DEBUG SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:32:07,832 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:32:07,833 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:32:23,003 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:32:23,003 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:32:35,259 DEBUG SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:32:38,164 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 10:32:38,165 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: stop_status
2022-02-01 10:32:42,989 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:32:44,990 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:32:48,993 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:32:52,987 DEBUG SenderThread:61045 [sender.py:send():234] send: telemetry
2022-02-01 10:32:52,988 DEBUG SenderThread:61045 [sender.py:send():234] send: exit
2022-02-01 10:32:52,988 INFO SenderThread:61045 [sender.py:send_exit():366] handling exit code: 0
2022-02-01 10:32:52,988 INFO SenderThread:61045 [sender.py:send_exit():368] handling runtime: 167
2022-02-01 10:32:52,989 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:52,990 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 10:32:52,990 INFO SenderThread:61045 [sender.py:send_exit():374] send defer
2022-02-01 10:32:52,990 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:52,991 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:52,992 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 0
2022-02-01 10:32:52,992 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:52,992 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 0
2022-02-01 10:32:52,992 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 1
2022-02-01 10:32:52,993 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:52,993 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 1
2022-02-01 10:32:52,996 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:32:52,996 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json
2022-02-01 10:32:53,088 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:53,088 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 1
2022-02-01 10:32:53,088 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 2
2022-02-01 10:32:53,088 DEBUG SenderThread:61045 [sender.py:send():234] send: stats
2022-02-01 10:32:53,089 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:53,089 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 2
2022-02-01 10:32:53,090 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:53,090 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 2
2022-02-01 10:32:53,090 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 3
2022-02-01 10:32:53,090 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:53,090 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 3
2022-02-01 10:32:53,091 DEBUG SenderThread:61045 [sender.py:send():234] send: summary
2022-02-01 10:32:53,092 INFO SenderThread:61045 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 10:32:53,092 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:53,092 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 3
2022-02-01 10:32:53,092 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 4
2022-02-01 10:32:53,093 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:53,093 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 4
2022-02-01 10:32:53,093 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:53,093 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 4
2022-02-01 10:32:53,103 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:53,344 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 5
2022-02-01 10:32:53,344 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:53,345 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:53,345 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 5
2022-02-01 10:32:53,345 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:53,345 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 5
2022-02-01 10:32:53,345 INFO SenderThread:61045 [dir_watcher.py:finish():283] shutting down directory watcher
2022-02-01 10:32:53,448 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:53,997 INFO Thread-8 :61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/config.yaml
2022-02-01 10:32:53,998 INFO SenderThread:61045 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json
2022-02-01 10:32:53,999 INFO SenderThread:61045 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files
2022-02-01 10:32:53,999 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log output.log
2022-02-01 10:32:54,000 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-metadata.json wandb-metadata.json
2022-02-01 10:32:54,001 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/config.yaml config.yaml
2022-02-01 10:32:54,007 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/conda-environment.yaml conda-environment.yaml
2022-02-01 10:32:54,014 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json wandb-summary.json
2022-02-01 10:32:54,015 INFO SenderThread:61045 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/requirements.txt requirements.txt
2022-02-01 10:32:54,019 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 6
2022-02-01 10:32:54,020 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:54,026 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:54,026 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 6
2022-02-01 10:32:54,026 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:54,027 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 6
2022-02-01 10:32:54,027 INFO SenderThread:61045 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 10:32:54,122 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:54,123 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:54,225 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:54,226 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:54,328 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:54,329 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:54,431 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:54,432 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:54,524 INFO Thread-14 :61045 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/wandb-summary.json
2022-02-01 10:32:54,531 INFO Thread-12 :61045 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/output.log
2022-02-01 10:32:54,534 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:54,534 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:54,542 INFO Thread-15 :61045 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/requirements.txt
2022-02-01 10:32:54,556 INFO Thread-13 :61045 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/files/config.yaml
2022-02-01 10:32:54,636 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:54,637 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:54,739 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:54,739 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:54,757 INFO Thread-7 :61045 [sender.py:transition_state():387] send defer: 7
2022-02-01 10:32:54,758 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:54,759 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 7
2022-02-01 10:32:54,759 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:54,759 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 7
2022-02-01 10:32:54,842 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:55,205 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 8
2022-02-01 10:32:55,206 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:55,206 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:55,207 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 8
2022-02-01 10:32:55,208 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:55,208 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 8
2022-02-01 10:32:55,208 INFO SenderThread:61045 [sender.py:transition_state():387] send defer: 9
2022-02-01 10:32:55,209 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: defer
2022-02-01 10:32:55,210 INFO HandlerThread:61045 [handler.py:handle_request_defer():147] handle defer: 9
2022-02-01 10:32:55,210 DEBUG SenderThread:61045 [sender.py:send():234] send: final
2022-02-01 10:32:55,210 DEBUG SenderThread:61045 [sender.py:send():234] send: footer
2022-02-01 10:32:55,210 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: defer
2022-02-01 10:32:55,210 INFO SenderThread:61045 [sender.py:send_request_defer():383] handle sender defer: 9
2022-02-01 10:32:55,309 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 10:32:55,310 DEBUG SenderThread:61045 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 10:32:55,310 INFO SenderThread:61045 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 10:32:55,613 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: get_summary
2022-02-01 10:32:55,615 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: sampled_history
2022-02-01 10:32:55,617 DEBUG HandlerThread:61045 [handler.py:handle_request():130] handle_request: shutdown
2022-02-01 10:32:55,617 INFO HandlerThread:61045 [handler.py:finish():731] shutting down handler
2022-02-01 10:32:56,210 INFO WriterThread:61045 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220201_103004-1yfj7vwy/run-1yfj7vwy.wandb
2022-02-01 10:32:56,611 INFO SenderThread:61045 [sender.py:finish():1070] shutting down sender
2022-02-01 10:32:56,611 INFO SenderThread:61045 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 10:32:56,611 INFO SenderThread:61045 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 10:32:56,615 INFO MainThread:61045 [internal.py:handle_exit():77] Internal process exited
|