File size: 17,305 Bytes
ef5fb31
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
2022-01-30 16:32:48,452 INFO    MainThread:52211 [internal.py:wandb_internal():87] W&B internal server running at pid: 52211, started at: 2022-01-30 16:32:48.452114
2022-01-30 16:32:48,456 INFO    WriterThread:52211 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/run-20w0bl60.wandb
2022-01-30 16:32:48,457 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: check_version
2022-01-30 16:32:48,460 DEBUG   SenderThread:52211 [sender.py:send():234] send: header
2022-01-30 16:32:48,460 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: check_version
2022-01-30 16:32:48,657 DEBUG   SenderThread:52211 [sender.py:send():234] send: run
2022-01-30 16:32:48,873 INFO    SenderThread:52211 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files
2022-01-30 16:32:48,873 INFO    SenderThread:52211 [sender.py:_start_run_threads():804] run started: 20w0bl60 with start time 1643560367
2022-01-30 16:32:48,873 DEBUG   SenderThread:52211 [sender.py:send():234] send: summary
2022-01-30 16:32:48,874 INFO    SenderThread:52211 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 16:32:48,876 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: run_start
2022-01-30 16:32:48,884 DEBUG   HandlerThread:52211 [meta.py:__init__():40] meta init
2022-01-30 16:32:48,885 DEBUG   HandlerThread:52211 [meta.py:__init__():54] meta init done
2022-01-30 16:32:48,885 DEBUG   HandlerThread:52211 [meta.py:probe():214] probe
2022-01-30 16:32:48,894 DEBUG   HandlerThread:52211 [meta.py:_setup_git():204] setup git
2022-01-30 16:32:48,923 DEBUG   HandlerThread:52211 [meta.py:_setup_git():211] setup git done
2022-01-30 16:32:48,923 DEBUG   HandlerThread:52211 [meta.py:_save_pip():58] save pip
2022-01-30 16:32:48,924 DEBUG   HandlerThread:52211 [meta.py:_save_pip():72] save pip done
2022-01-30 16:32:48,924 DEBUG   HandlerThread:52211 [meta.py:_save_conda():79] save conda
2022-01-30 16:32:49,449 DEBUG   HandlerThread:52211 [meta.py:_save_conda():89] save conda done
2022-01-30 16:32:49,450 DEBUG   HandlerThread:52211 [meta.py:probe():252] probe done
2022-01-30 16:32:49,458 DEBUG   SenderThread:52211 [sender.py:send():234] send: files
2022-01-30 16:32:49,459 INFO    SenderThread:52211 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-30 16:32:49,475 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 16:32:49,476 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: stop_status
2022-01-30 16:32:49,635 DEBUG   SenderThread:52211 [sender.py:send():234] send: config
2022-01-30 16:32:49,637 DEBUG   SenderThread:52211 [sender.py:send():234] send: metric
2022-01-30 16:32:49,639 DEBUG   SenderThread:52211 [sender.py:send():234] send: metric
2022-01-30 16:32:49,639 WARNING SenderThread:52211 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-01-30 16:32:49,876 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json
2022-01-30 16:32:49,877 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:49,877 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-metadata.json
2022-01-30 16:32:49,878 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/requirements.txt
2022-01-30 16:32:49,878 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/conda-environment.yaml
2022-01-30 16:32:50,007 INFO    Thread-11 :52211 [upload_job.py:push():137] Uploaded file /tmp/tmphz4zp49fwandb/20m3u1mt-wandb-metadata.json
2022-01-30 16:32:51,875 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:55,878 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:56,702 DEBUG   SenderThread:52211 [sender.py:send():234] send: telemetry
2022-01-30 16:32:56,703 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:56,703 DEBUG   SenderThread:52211 [sender.py:send():234] send: exit
2022-01-30 16:32:56,703 INFO    SenderThread:52211 [sender.py:send_exit():366] handling exit code: 1
2022-01-30 16:32:56,704 INFO    SenderThread:52211 [sender.py:send_exit():368] handling runtime: 7
2022-01-30 16:32:56,704 INFO    SenderThread:52211 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 16:32:56,705 INFO    SenderThread:52211 [sender.py:send_exit():374] send defer
2022-01-30 16:32:56,705 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:56,706 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,707 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-30 16:32:56,707 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,707 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-30 16:32:56,707 INFO    SenderThread:52211 [sender.py:transition_state():387] send defer: 1
2022-01-30 16:32:56,708 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,708 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-30 16:32:56,780 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,781 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-30 16:32:56,781 INFO    SenderThread:52211 [sender.py:transition_state():387] send defer: 2
2022-01-30 16:32:56,781 DEBUG   SenderThread:52211 [sender.py:send():234] send: stats
2022-01-30 16:32:56,782 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,782 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-30 16:32:56,783 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,783 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-30 16:32:56,783 INFO    SenderThread:52211 [sender.py:transition_state():387] send defer: 3
2022-01-30 16:32:56,784 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,784 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-30 16:32:56,784 DEBUG   SenderThread:52211 [sender.py:send():234] send: summary
2022-01-30 16:32:56,785 INFO    SenderThread:52211 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 16:32:56,785 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,785 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-30 16:32:56,785 INFO    SenderThread:52211 [sender.py:transition_state():387] send defer: 4
2022-01-30 16:32:56,786 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:56,786 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-30 16:32:56,787 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:56,787 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-30 16:32:56,808 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:56,879 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:56,879 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json
2022-01-30 16:32:57,074 INFO    SenderThread:52211 [sender.py:transition_state():387] send defer: 5
2022-01-30 16:32:57,074 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:57,075 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:57,076 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-30 16:32:57,076 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:57,076 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-30 16:32:57,076 INFO    SenderThread:52211 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-30 16:32:57,177 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:57,879 INFO    Thread-8  :52211 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/config.yaml
2022-01-30 16:32:57,881 INFO    SenderThread:52211 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files
2022-01-30 16:32:57,881 INFO    SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log output.log
2022-01-30 16:32:57,882 INFO    SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-metadata.json wandb-metadata.json
2022-01-30 16:32:57,882 INFO    SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/config.yaml config.yaml
2022-01-30 16:32:57,888 INFO    SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/conda-environment.yaml conda-environment.yaml
2022-01-30 16:32:57,896 INFO    SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json wandb-summary.json
2022-01-30 16:32:57,897 INFO    SenderThread:52211 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/requirements.txt requirements.txt
2022-01-30 16:32:57,904 INFO    SenderThread:52211 [sender.py:transition_state():387] send defer: 6
2022-01-30 16:32:57,905 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:57,907 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:57,912 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-30 16:32:57,913 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:57,915 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-30 16:32:57,915 INFO    SenderThread:52211 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 16:32:58,007 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,008 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,111 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,111 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,214 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,214 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,317 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,318 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,419 INFO    Thread-13 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/config.yaml
2022-01-30 16:32:58,424 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,427 INFO    Thread-12 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/output.log
2022-01-30 16:32:58,428 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,471 INFO    Thread-15 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/requirements.txt
2022-01-30 16:32:58,515 INFO    Thread-14 :52211 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/files/wandb-summary.json
2022-01-30 16:32:58,531 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,532 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,634 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,634 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,716 INFO    Thread-7  :52211 [sender.py:transition_state():387] send defer: 7
2022-01-30 16:32:58,717 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:58,717 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-30 16:32:58,718 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:58,718 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-30 16:32:58,736 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,862 INFO    SenderThread:52211 [sender.py:transition_state():387] send defer: 8
2022-01-30 16:32:58,862 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,863 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:58,864 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-30 16:32:58,864 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:58,864 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-30 16:32:58,864 INFO    SenderThread:52211 [sender.py:transition_state():387] send defer: 9
2022-01-30 16:32:58,866 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: defer
2022-01-30 16:32:58,866 DEBUG   SenderThread:52211 [sender.py:send():234] send: final
2022-01-30 16:32:58,866 INFO    HandlerThread:52211 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-30 16:32:58,866 DEBUG   SenderThread:52211 [sender.py:send():234] send: footer
2022-01-30 16:32:58,867 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: defer
2022-01-30 16:32:58,867 INFO    SenderThread:52211 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-30 16:32:58,965 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 16:32:58,966 DEBUG   SenderThread:52211 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 16:32:58,967 INFO    SenderThread:52211 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 16:32:59,277 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: get_summary
2022-01-30 16:32:59,279 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-30 16:32:59,280 DEBUG   HandlerThread:52211 [handler.py:handle_request():130] handle_request: shutdown
2022-01-30 16:32:59,280 INFO    HandlerThread:52211 [handler.py:finish():731] shutting down handler
2022-01-30 16:32:59,866 INFO    WriterThread:52211 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_163246-20w0bl60/run-20w0bl60.wandb
2022-01-30 16:33:00,275 INFO    SenderThread:52211 [sender.py:finish():1070] shutting down sender
2022-01-30 16:33:00,275 INFO    SenderThread:52211 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 16:33:00,275 INFO    SenderThread:52211 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 16:33:00,279 INFO    MainThread:52211 [internal.py:handle_exit():77] Internal process exited