File size: 20,163 Bytes
4641f67
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
cbeda52
6f51b2d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
2022-02-01 14:24:48,994 INFO    MainThread:29947 [internal.py:wandb_internal():87] W&B internal server running at pid: 29947, started at: 2022-02-01 14:24:48.994283
2022-02-01 14:24:48,998 INFO    WriterThread:29947 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/run-3hqii6h6.wandb
2022-02-01 14:24:48,999 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 14:24:49,002 DEBUG   SenderThread:29947 [sender.py:send():234] send: header
2022-02-01 14:24:49,002 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: check_version
2022-02-01 14:24:49,081 DEBUG   SenderThread:29947 [sender.py:send():234] send: run
2022-02-01 14:24:49,273 INFO    SenderThread:29947 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files
2022-02-01 14:24:49,273 INFO    SenderThread:29947 [sender.py:_start_run_threads():804] run started: 3hqii6h6 with start time 1643725488
2022-02-01 14:24:49,273 DEBUG   SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:24:49,274 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:24:49,276 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 14:24:49,335 DEBUG   HandlerThread:29947 [meta.py:__init__():40] meta init
2022-02-01 14:24:49,335 DEBUG   HandlerThread:29947 [meta.py:__init__():54] meta init done
2022-02-01 14:24:49,335 DEBUG   HandlerThread:29947 [meta.py:probe():214] probe
2022-02-01 14:24:49,342 DEBUG   HandlerThread:29947 [meta.py:_setup_git():204] setup git
2022-02-01 14:24:49,374 DEBUG   HandlerThread:29947 [meta.py:_setup_git():211] setup git done
2022-02-01 14:24:49,374 DEBUG   HandlerThread:29947 [meta.py:_save_pip():58] save pip
2022-02-01 14:24:49,375 DEBUG   HandlerThread:29947 [meta.py:_save_pip():72] save pip done
2022-02-01 14:24:49,375 DEBUG   HandlerThread:29947 [meta.py:_save_conda():79] save conda
2022-02-01 14:24:49,989 DEBUG   HandlerThread:29947 [meta.py:_save_conda():89] save conda done
2022-02-01 14:24:49,989 DEBUG   HandlerThread:29947 [meta.py:probe():252] probe done
2022-02-01 14:24:49,998 DEBUG   SenderThread:29947 [sender.py:send():234] send: files
2022-02-01 14:24:49,999 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 14:24:50,010 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:24:50,011 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:24:50,173 DEBUG   SenderThread:29947 [sender.py:send():234] send: config
2022-02-01 14:24:50,176 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,176 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 WARNING SenderThread:29947 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,177 DEBUG   SenderThread:29947 [sender.py:send():234] send: metric
2022-02-01 14:24:50,178 DEBUG   SenderThread:29947 [sender.py:send():234] send: history
2022-02-01 14:24:50,178 DEBUG   SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:24:50,178 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:24:50,276 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:24:50,277 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/requirements.txt
2022-02-01 14:24:50,277 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/conda-environment.yaml
2022-02-01 14:24:50,278 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-metadata.json
2022-02-01 14:24:50,278 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json
2022-02-01 14:24:50,617 INFO    Thread-11 :29947 [upload_job.py:push():137] Uploaded file /tmp/tmpp56x1syiwandb/1qh281ys-wandb-metadata.json
2022-02-01 14:24:52,275 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:24:54,276 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:25:05,175 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:05,176 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:17,592 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:25:20,294 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml
2022-02-01 14:25:20,335 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:20,335 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:35,486 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:35,487 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:25:47,927 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:25:50,662 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:25:50,663 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:26:05,830 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 14:26:05,830 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: stop_status
2022-02-01 14:26:12,326 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:16,328 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:18,207 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:26:18,330 INFO    Thread-8  :29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:23,406 DEBUG   SenderThread:29947 [sender.py:send():234] send: telemetry
2022-02-01 14:26:23,407 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:23,407 DEBUG   SenderThread:29947 [sender.py:send():234] send: exit
2022-02-01 14:26:23,407 INFO    SenderThread:29947 [sender.py:send_exit():366] handling exit code: 0
2022-02-01 14:26:23,407 INFO    SenderThread:29947 [sender.py:send_exit():368] handling runtime: 94
2022-02-01 14:26:23,411 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:26:23,411 INFO    SenderThread:29947 [sender.py:send_exit():374] send defer
2022-02-01 14:26:23,412 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:23,412 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,413 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 0
2022-02-01 14:26:23,413 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,413 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 0
2022-02-01 14:26:23,413 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 1
2022-02-01 14:26:23,414 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,414 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 1
2022-02-01 14:26:23,486 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,486 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 1
2022-02-01 14:26:23,486 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 2
2022-02-01 14:26:23,486 DEBUG   SenderThread:29947 [sender.py:send():234] send: stats
2022-02-01 14:26:23,487 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,487 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 2
2022-02-01 14:26:23,488 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,488 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 2
2022-02-01 14:26:23,488 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 3
2022-02-01 14:26:23,488 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,488 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 3
2022-02-01 14:26:23,489 DEBUG   SenderThread:29947 [sender.py:send():234] send: summary
2022-02-01 14:26:23,490 INFO    SenderThread:29947 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 14:26:23,490 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,490 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 3
2022-02-01 14:26:23,490 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 4
2022-02-01 14:26:23,491 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,491 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 4
2022-02-01 14:26:23,491 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,491 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 4
2022-02-01 14:26:23,515 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:23,724 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 5
2022-02-01 14:26:23,724 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:23,726 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:23,726 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 5
2022-02-01 14:26:23,726 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:23,726 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 5
2022-02-01 14:26:23,726 INFO    SenderThread:29947 [dir_watcher.py:finish():283] shutting down directory watcher
2022-02-01 14:26:23,827 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,332 INFO    SenderThread:29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:24,333 INFO    SenderThread:29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json
2022-02-01 14:26:24,334 INFO    SenderThread:29947 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml
2022-02-01 14:26:24,334 INFO    SenderThread:29947 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files
2022-02-01 14:26:24,334 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log output.log
2022-02-01 14:26:24,335 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-metadata.json wandb-metadata.json
2022-02-01 14:26:24,335 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml config.yaml
2022-02-01 14:26:24,341 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/conda-environment.yaml conda-environment.yaml
2022-02-01 14:26:24,348 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json wandb-summary.json
2022-02-01 14:26:24,348 INFO    SenderThread:29947 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/requirements.txt requirements.txt
2022-02-01 14:26:24,349 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 6
2022-02-01 14:26:24,350 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,357 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:24,357 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 6
2022-02-01 14:26:24,357 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:24,358 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 6
2022-02-01 14:26:24,358 INFO    SenderThread:29947 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 14:26:24,452 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,453 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,555 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,560 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,667 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,668 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,770 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,771 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,873 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,874 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:24,888 INFO    Thread-12 :29947 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/output.log
2022-02-01 14:26:24,897 INFO    Thread-13 :29947 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/config.yaml
2022-02-01 14:26:24,913 INFO    Thread-15 :29947 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/requirements.txt
2022-02-01 14:26:24,919 INFO    Thread-14 :29947 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/files/wandb-summary.json
2022-02-01 14:26:24,976 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:24,976 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:25,079 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:25,079 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:25,120 INFO    Thread-7  :29947 [sender.py:transition_state():387] send defer: 7
2022-02-01 14:26:25,122 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:25,122 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 7
2022-02-01 14:26:25,122 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:25,122 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 7
2022-02-01 14:26:25,182 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:25,552 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 8
2022-02-01 14:26:25,553 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:25,554 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:25,554 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 8
2022-02-01 14:26:25,555 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:25,555 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 8
2022-02-01 14:26:25,555 INFO    SenderThread:29947 [sender.py:transition_state():387] send defer: 9
2022-02-01 14:26:25,556 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: defer
2022-02-01 14:26:25,557 INFO    HandlerThread:29947 [handler.py:handle_request_defer():147] handle defer: 9
2022-02-01 14:26:25,557 DEBUG   SenderThread:29947 [sender.py:send():234] send: final
2022-02-01 14:26:25,557 DEBUG   SenderThread:29947 [sender.py:send():234] send: footer
2022-02-01 14:26:25,557 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: defer
2022-02-01 14:26:25,557 INFO    SenderThread:29947 [sender.py:send_request_defer():383] handle sender defer: 9
2022-02-01 14:26:25,656 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 14:26:25,656 DEBUG   SenderThread:29947 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 14:26:25,657 INFO    SenderThread:29947 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 14:26:25,966 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: get_summary
2022-02-01 14:26:25,968 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: sampled_history
2022-02-01 14:26:25,970 DEBUG   HandlerThread:29947 [handler.py:handle_request():130] handle_request: shutdown
2022-02-01 14:26:25,970 INFO    HandlerThread:29947 [handler.py:finish():731] shutting down handler
2022-02-01 14:26:26,557 INFO    WriterThread:29947 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220201_142447-3hqii6h6/run-3hqii6h6.wandb
2022-02-01 14:26:26,963 INFO    SenderThread:29947 [sender.py:finish():1070] shutting down sender
2022-02-01 14:26:26,964 INFO    SenderThread:29947 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 14:26:26,964 INFO    SenderThread:29947 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 14:26:26,967 INFO    MainThread:29947 [internal.py:handle_exit():77] Internal process exited