File size: 29,455 Bytes
ef5fb31
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
2022-01-30 17:54:18,493 INFO    MainThread:22568 [internal.py:wandb_internal():87] W&B internal server running at pid: 22568, started at: 2022-01-30 17:54:18.493348
2022-01-30 17:54:18,496 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: check_version
2022-01-30 17:54:18,497 INFO    WriterThread:22568 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/run-3n44neww.wandb
2022-01-30 17:54:18,498 DEBUG   SenderThread:22568 [sender.py:send():234] send: header
2022-01-30 17:54:18,498 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: check_version
2022-01-30 17:54:18,588 DEBUG   SenderThread:22568 [sender.py:send():234] send: run
2022-01-30 17:54:18,778 INFO    SenderThread:22568 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files
2022-01-30 17:54:18,779 INFO    SenderThread:22568 [sender.py:_start_run_threads():804] run started: 3n44neww with start time 1643565257
2022-01-30 17:54:18,779 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:54:18,779 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:54:18,781 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: run_start
2022-01-30 17:54:18,803 DEBUG   HandlerThread:22568 [meta.py:__init__():40] meta init
2022-01-30 17:54:18,803 DEBUG   HandlerThread:22568 [meta.py:__init__():54] meta init done
2022-01-30 17:54:18,803 DEBUG   HandlerThread:22568 [meta.py:probe():214] probe
2022-01-30 17:54:18,814 DEBUG   HandlerThread:22568 [meta.py:_setup_git():204] setup git
2022-01-30 17:54:18,844 DEBUG   HandlerThread:22568 [meta.py:_setup_git():211] setup git done
2022-01-30 17:54:18,844 DEBUG   HandlerThread:22568 [meta.py:_save_pip():58] save pip
2022-01-30 17:54:18,845 DEBUG   HandlerThread:22568 [meta.py:_save_pip():72] save pip done
2022-01-30 17:54:18,845 DEBUG   HandlerThread:22568 [meta.py:_save_conda():79] save conda
2022-01-30 17:54:19,393 DEBUG   HandlerThread:22568 [meta.py:_save_conda():89] save conda done
2022-01-30 17:54:19,394 DEBUG   HandlerThread:22568 [meta.py:probe():252] probe done
2022-01-30 17:54:19,402 DEBUG   SenderThread:22568 [sender.py:send():234] send: files
2022-01-30 17:54:19,402 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-30 17:54:19,413 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:54:19,414 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:54:19,575 DEBUG   SenderThread:22568 [sender.py:send():234] send: config
2022-01-30 17:54:19,578 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:19,578 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:19,578 WARNING SenderThread:22568 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-01-30 17:54:19,782 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:54:19,782 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:19,783 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/requirements.txt
2022-01-30 17:54:19,783 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/conda-environment.yaml
2022-01-30 17:54:19,783 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-metadata.json
2022-01-30 17:54:19,939 INFO    Thread-11 :22568 [upload_job.py:push():137] Uploaded file /tmp/tmpzu6sqwh8wandb/1uonnopl-wandb-metadata.json
2022-01-30 17:54:21,780 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:27,784 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:29,786 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:31,787 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:34,745 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:54:34,745 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:54:37,791 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:39,019 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:39,020 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:39,020 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:39,021 DEBUG   SenderThread:22568 [sender.py:send():234] send: history
2022-01-30 17:54:39,021 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:54:39,022 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:54:39,792 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:54:39,793 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:41,854 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:41,854 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:41,855 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:41,855 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:41,855 DEBUG   SenderThread:22568 [sender.py:send():234] send: metric
2022-01-30 17:54:41,855 DEBUG   SenderThread:22568 [sender.py:send():234] send: history
2022-01-30 17:54:41,856 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:54:41,861 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:54:42,794 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:54:43,795 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:45,797 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:47,048 DEBUG   SenderThread:22568 [sender.py:send():234] send: stats
2022-01-30 17:54:49,799 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:49,800 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/config.yaml
2022-01-30 17:54:49,989 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:54:49,989 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:54:53,802 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:56,803 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:54:59,567 DEBUG   SenderThread:22568 [sender.py:send():234] send: history
2022-01-30 17:54:59,568 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:54:59,571 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:54:59,805 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:55:01,807 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:02,327 DEBUG   SenderThread:22568 [sender.py:send():234] send: history
2022-01-30 17:55:02,328 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:55:02,329 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:55:02,807 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:55:02,808 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:03,808 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:04,809 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:05,223 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:55:05,224 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:55:06,810 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:10,813 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:14,815 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:17,371 DEBUG   SenderThread:22568 [sender.py:send():234] send: stats
2022-01-30 17:55:18,522 DEBUG   SenderThread:22568 [sender.py:send():234] send: history
2022-01-30 17:55:18,523 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:55:18,524 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:55:18,817 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:55:18,818 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:19,818 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:20,385 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:55:20,386 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:55:20,819 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:21,292 DEBUG   SenderThread:22568 [sender.py:send():234] send: history
2022-01-30 17:55:21,293 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:55:21,294 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:55:21,819 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:55:22,820 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:26,823 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:32,827 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:34,828 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:35,544 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:55:35,544 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:55:36,830 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:41,459 DEBUG   SenderThread:22568 [sender.py:send():234] send: history
2022-01-30 17:55:41,459 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:55:41,460 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:55:41,833 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:55:42,834 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:44,141 DEBUG   SenderThread:22568 [sender.py:send():234] send: history
2022-01-30 17:55:44,142 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:55:44,143 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:55:44,835 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:55:44,836 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:46,836 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:47,657 DEBUG   SenderThread:22568 [sender.py:send():234] send: stats
2022-01-30 17:55:48,838 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:50,808 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:55:50,809 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:55:50,839 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:51,470 WARNING MainThread:22568 [internal.py:wandb_internal():146] Internal process interrupt: 1
2022-01-30 17:55:52,841 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:54,071 DEBUG   SenderThread:22568 [sender.py:send():234] send: telemetry
2022-01-30 17:55:54,072 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:54,072 DEBUG   SenderThread:22568 [sender.py:send():234] send: exit
2022-01-30 17:55:54,073 INFO    SenderThread:22568 [sender.py:send_exit():366] handling exit code: 255
2022-01-30 17:55:54,073 INFO    SenderThread:22568 [sender.py:send_exit():368] handling runtime: 95
2022-01-30 17:55:54,074 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:55:54,074 INFO    SenderThread:22568 [sender.py:send_exit():374] send defer
2022-01-30 17:55:54,075 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:54,075 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:54,076 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-30 17:55:54,077 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:54,077 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-30 17:55:54,077 INFO    SenderThread:22568 [sender.py:transition_state():387] send defer: 1
2022-01-30 17:55:54,077 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:54,078 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-30 17:55:54,104 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:54,104 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-30 17:55:54,104 INFO    SenderThread:22568 [sender.py:transition_state():387] send defer: 2
2022-01-30 17:55:54,105 DEBUG   SenderThread:22568 [sender.py:send():234] send: stats
2022-01-30 17:55:54,106 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:54,106 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-30 17:55:54,106 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:54,106 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-30 17:55:54,106 INFO    SenderThread:22568 [sender.py:transition_state():387] send defer: 3
2022-01-30 17:55:54,107 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:54,107 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-30 17:55:54,107 DEBUG   SenderThread:22568 [sender.py:send():234] send: summary
2022-01-30 17:55:54,108 INFO    SenderThread:22568 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:55:54,108 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:54,108 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-30 17:55:54,109 INFO    SenderThread:22568 [sender.py:transition_state():387] send defer: 4
2022-01-30 17:55:54,109 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:54,109 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-30 17:55:54,109 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:54,109 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-30 17:55:54,178 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:54,344 INFO    SenderThread:22568 [sender.py:transition_state():387] send defer: 5
2022-01-30 17:55:54,344 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:54,346 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:54,346 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-30 17:55:54,346 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:54,346 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-30 17:55:54,346 INFO    SenderThread:22568 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-30 17:55:54,448 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:54,842 INFO    Thread-8  :22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:55:54,844 INFO    SenderThread:22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:54,844 INFO    SenderThread:22568 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/config.yaml
2022-01-30 17:55:54,845 INFO    SenderThread:22568 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files
2022-01-30 17:55:54,845 INFO    SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log output.log
2022-01-30 17:55:54,845 INFO    SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-metadata.json wandb-metadata.json
2022-01-30 17:55:54,846 INFO    SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/config.yaml config.yaml
2022-01-30 17:55:54,846 INFO    SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/conda-environment.yaml conda-environment.yaml
2022-01-30 17:55:54,847 INFO    SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json wandb-summary.json
2022-01-30 17:55:54,860 INFO    SenderThread:22568 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/requirements.txt requirements.txt
2022-01-30 17:55:54,868 INFO    SenderThread:22568 [sender.py:transition_state():387] send defer: 6
2022-01-30 17:55:54,869 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:54,878 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:54,879 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-30 17:55:54,879 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:54,880 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-30 17:55:54,880 INFO    SenderThread:22568 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 17:55:54,973 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:54,974 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:55,077 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:55,078 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:55,181 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:55,182 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:55,286 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:55,286 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:55,368 INFO    Thread-13 :22568 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/config.yaml
2022-01-30 17:55:55,370 INFO    Thread-14 :22568 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/wandb-summary.json
2022-01-30 17:55:55,373 INFO    Thread-15 :22568 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/requirements.txt
2022-01-30 17:55:55,390 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:55,390 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:55,400 INFO    Thread-12 :22568 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/files/output.log
2022-01-30 17:55:55,493 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:55,493 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:55,596 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:55,597 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:55,601 INFO    Thread-7  :22568 [sender.py:transition_state():387] send defer: 7
2022-01-30 17:55:55,602 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:55,602 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-30 17:55:55,602 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:55,602 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-30 17:55:55,699 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:56,187 INFO    SenderThread:22568 [sender.py:transition_state():387] send defer: 8
2022-01-30 17:55:56,187 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:56,188 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:56,189 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-30 17:55:56,189 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:56,189 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-30 17:55:56,189 INFO    SenderThread:22568 [sender.py:transition_state():387] send defer: 9
2022-01-30 17:55:56,191 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:55:56,191 INFO    HandlerThread:22568 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-30 17:55:56,191 DEBUG   SenderThread:22568 [sender.py:send():234] send: final
2022-01-30 17:55:56,191 DEBUG   SenderThread:22568 [sender.py:send():234] send: footer
2022-01-30 17:55:56,192 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: defer
2022-01-30 17:55:56,192 INFO    SenderThread:22568 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-30 17:55:56,290 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:55:56,291 DEBUG   SenderThread:22568 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:55:56,291 INFO    SenderThread:22568 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 17:55:56,600 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: get_summary
2022-01-30 17:55:56,602 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-30 17:55:56,604 DEBUG   HandlerThread:22568 [handler.py:handle_request():130] handle_request: shutdown
2022-01-30 17:55:56,604 INFO    HandlerThread:22568 [handler.py:finish():731] shutting down handler
2022-01-30 17:55:57,191 INFO    WriterThread:22568 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_175417-3n44neww/run-3n44neww.wandb
2022-01-30 17:55:57,598 INFO    SenderThread:22568 [sender.py:finish():1070] shutting down sender
2022-01-30 17:55:57,598 INFO    SenderThread:22568 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 17:55:57,598 INFO    SenderThread:22568 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 17:55:57,602 INFO    MainThread:22568 [internal.py:handle_exit():77] Internal process exited