File size: 17,305 Bytes
ef5fb31 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 |
2022-01-30 16:32:48,452 INFO MainThread:52211 [internal.py:wandb_internal():87] W&B internal server running at pid: 52211, started at: 2022-01-30 16:32:48.452114
2022-01-30 16:32:48,456 INFO WriterThread:52211 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/run-20w0bl60.wandb
2022-01-30 16:32:48,457 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: check_version
2022-01-30 16:32:48,460 DEBUG SenderThread:52211 [sender.py:send():234] send: header
2022-01-30 16:32:48,460 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: check_version
2022-01-30 16:32:48,657 DEBUG SenderThread:52211 [sender.py:send():234] send: run
2022-01-30 16:32:48,873 INFO SenderThread:52211 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files
2022-01-30 16:32:48,873 INFO SenderThread:52211 [sender.py:_start_run_threads():804] run started: 20w0bl60 with start time 1643560367
2022-01-30 16:32:48,873 DEBUG SenderThread:52211 [sender.py:send():234] send: summary
2022-01-30 16:32:48,874 INFO SenderThread:52211 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 16:32:48,876 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: run_start
2022-01-30 16:32:48,884 DEBUG HandlerThread:52211 [meta.py:__init__():40] meta init
2022-01-30 16:32:48,885 DEBUG HandlerThread:52211 [meta.py:__init__():54] meta init done
2022-01-30 16:32:48,885 DEBUG HandlerThread:52211 [meta.py:probe():214] probe
2022-01-30 16:32:48,894 DEBUG HandlerThread:52211 [meta.py:_setup_git():204] setup git
2022-01-30 16:32:48,923 DEBUG HandlerThread:52211 [meta.py:_setup_git():211] setup git done
2022-01-30 16:32:48,923 DEBUG HandlerThread:52211 [meta.py:_save_pip():58] save pip
2022-01-30 16:32:48,924 DEBUG HandlerThread:52211 [meta.py:_save_pip():72] save pip done
2022-01-30 16:32:48,924 DEBUG HandlerThread:52211 [meta.py:_save_conda():79] save conda
2022-01-30 16:32:49,449 DEBUG HandlerThread:52211 [meta.py:_save_conda():89] save conda done
2022-01-30 16:32:49,450 DEBUG HandlerThread:52211 [meta.py:probe():252] probe done
2022-01-30 16:32:49,458 DEBUG SenderThread:52211 [sender.py:send():234] send: files
2022-01-30 16:32:49,459 INFO SenderThread:52211 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-30 16:32:49,475 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 16:32:49,476 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: stop_status
2022-01-30 16:32:49,635 DEBUG SenderThread:52211 [sender.py:send():234] send: config
2022-01-30 16:32:49,637 DEBUG SenderThread:52211 [sender.py:send():234] send: metric
2022-01-30 16:32:49,639 DEBUG SenderThread:52211 [sender.py:send():234] send: metric
2022-01-30 16:32:49,639 WARNING SenderThread:52211 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-01-30 16:32:49,876 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json
2022-01-30 16:32:49,877 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:49,877 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-metadata.json
2022-01-30 16:32:49,878 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/requirements.txt
2022-01-30 16:32:49,878 INFO Thread-8 :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/conda-environment.yaml
2022-01-30 16:32:50,007 INFO Thread-11 :52211 [upload_job.py:push():137] Uploaded file /tmp/tmphz4zp49fwandb/20m3u1mt-wandb-metadata.json
2022-01-30 16:32:51,875 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:55,878 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:56,702 DEBUG SenderThread:52211 [sender.py:send():234] send: telemetry
2022-01-30 16:32:56,703 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:56,703 DEBUG SenderThread:52211 [sender.py:send():234] send: exit
2022-01-30 16:32:56,703 INFO SenderThread:52211 [sender.py:send_exit():366] handling exit code: 1
2022-01-30 16:32:56,704 INFO SenderThread:52211 [sender.py:send_exit():368] handling runtime: 7
2022-01-30 16:32:56,704 INFO SenderThread:52211 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 16:32:56,705 INFO SenderThread:52211 [sender.py:send_exit():374] send defer
2022-01-30 16:32:56,705 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:56,706 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,707 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-30 16:32:56,707 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,707 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-30 16:32:56,707 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 1
2022-01-30 16:32:56,708 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,708 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-30 16:32:56,780 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,781 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-30 16:32:56,781 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 2
2022-01-30 16:32:56,781 DEBUG SenderThread:52211 [sender.py:send():234] send: stats
2022-01-30 16:32:56,782 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,782 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-30 16:32:56,783 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,783 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-30 16:32:56,783 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 3
2022-01-30 16:32:56,784 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,784 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-30 16:32:56,784 DEBUG SenderThread:52211 [sender.py:send():234] send: summary
2022-01-30 16:32:56,785 INFO SenderThread:52211 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 16:32:56,785 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,785 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-30 16:32:56,785 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 4
2022-01-30 16:32:56,786 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,786 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-30 16:32:56,787 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,787 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-30 16:32:56,808 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:56,879 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:56,879 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json
2022-01-30 16:32:57,074 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 5
2022-01-30 16:32:57,074 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:57,075 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:57,076 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-30 16:32:57,076 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:57,076 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-30 16:32:57,076 INFO SenderThread:52211 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-30 16:32:57,177 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:57,879 INFO Thread-8 :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/config.yaml
2022-01-30 16:32:57,881 INFO SenderThread:52211 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files
2022-01-30 16:32:57,881 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log output.log
2022-01-30 16:32:57,882 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-metadata.json wandb-metadata.json
2022-01-30 16:32:57,882 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/config.yaml config.yaml
2022-01-30 16:32:57,888 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/conda-environment.yaml conda-environment.yaml
2022-01-30 16:32:57,896 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json wandb-summary.json
2022-01-30 16:32:57,897 INFO SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/requirements.txt requirements.txt
2022-01-30 16:32:57,904 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 6
2022-01-30 16:32:57,905 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:57,907 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:57,912 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-30 16:32:57,913 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:57,915 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-30 16:32:57,915 INFO SenderThread:52211 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 16:32:58,007 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,008 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,111 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,111 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,214 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,214 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,317 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,318 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,419 INFO Thread-13 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/config.yaml
2022-01-30 16:32:58,424 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,427 INFO Thread-12 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:58,428 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,471 INFO Thread-15 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/requirements.txt
2022-01-30 16:32:58,515 INFO Thread-14 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json
2022-01-30 16:32:58,531 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,532 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,634 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,634 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,716 INFO Thread-7 :52211 [sender.py:transition_state():387] send defer: 7
2022-01-30 16:32:58,717 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:58,717 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-30 16:32:58,718 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:58,718 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-30 16:32:58,736 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,862 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 8
2022-01-30 16:32:58,862 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,863 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:58,864 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-30 16:32:58,864 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:58,864 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-30 16:32:58,864 INFO SenderThread:52211 [sender.py:transition_state():387] send defer: 9
2022-01-30 16:32:58,866 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:58,866 DEBUG SenderThread:52211 [sender.py:send():234] send: final
2022-01-30 16:32:58,866 INFO HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-30 16:32:58,866 DEBUG SenderThread:52211 [sender.py:send():234] send: footer
2022-01-30 16:32:58,867 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:58,867 INFO SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-30 16:32:58,965 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,966 DEBUG SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,967 INFO SenderThread:52211 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 16:32:59,277 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: get_summary
2022-01-30 16:32:59,279 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-30 16:32:59,280 DEBUG HandlerThread:52211 [handler.py:handle_request():130] handle_request: shutdown
2022-01-30 16:32:59,280 INFO HandlerThread:52211 [handler.py:finish():731] shutting down handler
2022-01-30 16:32:59,866 INFO WriterThread:52211 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/run-20w0bl60.wandb
2022-01-30 16:33:00,275 INFO SenderThread:52211 [sender.py:finish():1070] shutting down sender
2022-01-30 16:33:00,275 INFO SenderThread:52211 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 16:33:00,275 INFO SenderThread:52211 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 16:33:00,279 INFO MainThread:52211 [internal.py:handle_exit():77] Internal process exited
|