File size: 19,167 Bytes
981dcc0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
2022-02-01 15:08:19,400 INFO    MainThread:3248864 [internal.py:wandb_internal():87] W&B internal server running at pid: 3248864, started at: 2022-02-01 15:08:19.399884
2022-02-01 15:08:19,402 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: check_version
2022-02-01 15:08:19,402 INFO    WriterThread:3248864 [datastore.py:open_for_write():77] open: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb
2022-02-01 15:08:19,405 DEBUG   SenderThread:3248864 [sender.py:send():234] send: header
2022-02-01 15:08:19,405 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: check_version
2022-02-01 15:08:19,562 DEBUG   SenderThread:3248864 [sender.py:send():234] send: run
2022-02-01 15:08:19,847 INFO    SenderThread:3248864 [dir_watcher.py:__init__():169] watching files in: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files
2022-02-01 15:08:19,848 INFO    SenderThread:3248864 [sender.py:_start_run_threads():804] run started: 33gv5m8t with start time 1643724498
2022-02-01 15:08:19,848 DEBUG   SenderThread:3248864 [sender.py:send():234] send: summary
2022-02-01 15:08:19,848 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:19,850 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: run_start
2022-02-01 15:08:20,854 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:21,089 DEBUG   HandlerThread:3248864 [meta.py:__init__():40] meta init
2022-02-01 15:08:21,090 DEBUG   HandlerThread:3248864 [meta.py:__init__():54] meta init done
2022-02-01 15:08:21,090 DEBUG   HandlerThread:3248864 [meta.py:probe():214] probe
2022-02-01 15:08:21,096 DEBUG   HandlerThread:3248864 [meta.py:_setup_git():204] setup git
2022-02-01 15:08:21,129 DEBUG   HandlerThread:3248864 [meta.py:_setup_git():211] setup git done
2022-02-01 15:08:21,130 DEBUG   HandlerThread:3248864 [meta.py:_save_code():92] save code
2022-02-01 15:08:21,144 DEBUG   HandlerThread:3248864 [meta.py:_save_code():113] save code done
2022-02-01 15:08:21,144 DEBUG   HandlerThread:3248864 [meta.py:_save_patches():130] save patches
2022-02-01 15:08:21,223 DEBUG   HandlerThread:3248864 [meta.py:_save_patches():172] save patches done
2022-02-01 15:08:21,223 DEBUG   HandlerThread:3248864 [meta.py:_save_pip():58] save pip
2022-02-01 15:08:21,224 DEBUG   HandlerThread:3248864 [meta.py:_save_pip():72] save pip done
2022-02-01 15:08:21,224 DEBUG   HandlerThread:3248864 [meta.py:probe():252] probe done
2022-02-01 15:08:21,230 DEBUG   SenderThread:3248864 [sender.py:send():234] send: files
2022-02-01 15:08:21,231 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-02-01 15:08:21,232 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file code/run_speech_recognition_ctc.py with policy now
2022-02-01 15:08:21,240 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: stop_status
2022-02-01 15:08:21,245 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: stop_status
2022-02-01 15:08:21,493 DEBUG   SenderThread:3248864 [sender.py:send():234] send: config
2022-02-01 15:08:21,495 DEBUG   SenderThread:3248864 [sender.py:send():234] send: metric
2022-02-01 15:08:21,496 DEBUG   SenderThread:3248864 [sender.py:send():234] send: metric
2022-02-01 15:08:21,496 WARNING SenderThread:3248864 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-02-01 15:08:21,852 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json
2022-02-01 15:08:21,852 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py
2022-02-01 15:08:21,853 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt
2022-02-01 15:08:21,853 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:21,853 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_created():217] file/dir created: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code
2022-02-01 15:08:21,960 INFO    Thread-11 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/bkne7cxj-wandb-metadata.json
2022-02-01 15:08:21,969 INFO    Thread-12 :3248864 [upload_job.py:push():137] Uploaded file /tmp/tmpxu9hw9m_wandb/1l3kkiz5-code/run_speech_recognition_ctc.py
2022-02-01 15:08:23,853 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:25,854 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:25,932 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:25,933 DEBUG   SenderThread:3248864 [sender.py:send():234] send: telemetry
2022-02-01 15:08:25,933 DEBUG   SenderThread:3248864 [sender.py:send():234] send: exit
2022-02-01 15:08:25,933 INFO    SenderThread:3248864 [sender.py:send_exit():366] handling exit code: 1
2022-02-01 15:08:25,934 INFO    SenderThread:3248864 [sender.py:send_exit():368] handling runtime: 6
2022-02-01 15:08:25,934 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:25,934 INFO    SenderThread:3248864 [sender.py:send_exit():374] send defer
2022-02-01 15:08:25,935 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:25,936 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:25,936 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 0
2022-02-01 15:08:25,936 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:25,936 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 0
2022-02-01 15:08:25,936 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 1
2022-02-01 15:08:25,937 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:25,937 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 1
2022-02-01 15:08:26,020 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,020 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 1
2022-02-01 15:08:26,020 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 2
2022-02-01 15:08:26,021 DEBUG   SenderThread:3248864 [sender.py:send():234] send: stats
2022-02-01 15:08:26,022 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,022 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 2
2022-02-01 15:08:26,022 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,022 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 2
2022-02-01 15:08:26,022 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 3
2022-02-01 15:08:26,023 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,023 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 3
2022-02-01 15:08:26,023 DEBUG   SenderThread:3248864 [sender.py:send():234] send: summary
2022-02-01 15:08:26,024 INFO    SenderThread:3248864 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-02-01 15:08:26,024 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,024 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 3
2022-02-01 15:08:26,024 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 4
2022-02-01 15:08:26,025 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,025 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 4
2022-02-01 15:08:26,025 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,026 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 4
2022-02-01 15:08:26,038 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,367 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 5
2022-02-01 15:08:26,367 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:26,368 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,368 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 5
2022-02-01 15:08:26,368 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,368 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 5
2022-02-01 15:08:26,368 INFO    SenderThread:3248864 [dir_watcher.py:finish():283] shutting down directory watcher
2022-02-01 15:08:26,469 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,854 INFO    Thread-8  :3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:26,855 INFO    SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml
2022-02-01 15:08:26,856 INFO    SenderThread:3248864 [dir_watcher.py:_on_file_modified():230] file/dir modified: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:26,856 INFO    SenderThread:3248864 [dir_watcher.py:finish():313] scan: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files
2022-02-01 15:08:26,856 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml config.yaml
2022-02-01 15:08:26,856 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt requirements.txt
2022-02-01 15:08:26,857 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json wandb-summary.json
2022-02-01 15:08:26,857 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log output.log
2022-02-01 15:08:26,857 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-metadata.json wandb-metadata.json
2022-02-01 15:08:26,869 INFO    SenderThread:3248864 [dir_watcher.py:finish():327] scan save: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/code/run_speech_recognition_ctc.py code/run_speech_recognition_ctc.py
2022-02-01 15:08:26,869 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 6
2022-02-01 15:08:26,869 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:26,881 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:26,881 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 6
2022-02-01 15:08:26,882 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:26,882 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 6
2022-02-01 15:08:26,882 INFO    SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 15:08:26,984 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:26,984 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,086 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,087 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,189 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,189 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,291 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,292 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,394 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,394 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,497 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,497 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,545 INFO    Thread-13 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/config.yaml
2022-02-01 15:08:27,551 INFO    Thread-14 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/requirements.txt
2022-02-01 15:08:27,553 INFO    Thread-16 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/output.log
2022-02-01 15:08:27,565 INFO    Thread-15 :3248864 [upload_job.py:push():137] Uploaded file /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/files/wandb-summary.json
2022-02-01 15:08:27,599 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,600 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,702 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:27,702 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:27,766 INFO    Thread-7  :3248864 [sender.py:transition_state():387] send defer: 7
2022-02-01 15:08:27,767 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:27,767 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 7
2022-02-01 15:08:27,767 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:27,767 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 7
2022-02-01 15:08:27,805 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:28,272 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 8
2022-02-01 15:08:28,272 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:28,273 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:28,273 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 8
2022-02-01 15:08:28,274 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:28,274 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 8
2022-02-01 15:08:28,274 INFO    SenderThread:3248864 [sender.py:transition_state():387] send defer: 9
2022-02-01 15:08:28,275 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: defer
2022-02-01 15:08:28,275 DEBUG   SenderThread:3248864 [sender.py:send():234] send: final
2022-02-01 15:08:28,275 INFO    HandlerThread:3248864 [handler.py:handle_request_defer():147] handle defer: 9
2022-02-01 15:08:28,276 DEBUG   SenderThread:3248864 [sender.py:send():234] send: footer
2022-02-01 15:08:28,276 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: defer
2022-02-01 15:08:28,276 INFO    SenderThread:3248864 [sender.py:send_request_defer():383] handle sender defer: 9
2022-02-01 15:08:28,375 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: poll_exit
2022-02-01 15:08:28,375 DEBUG   SenderThread:3248864 [sender.py:send_request():248] send_request: poll_exit
2022-02-01 15:08:28,375 INFO    SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 15:08:28,858 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: get_summary
2022-02-01 15:08:28,859 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: sampled_history
2022-02-01 15:08:28,861 DEBUG   HandlerThread:3248864 [handler.py:handle_request():130] handle_request: shutdown
2022-02-01 15:08:28,861 INFO    HandlerThread:3248864 [handler.py:finish():731] shutting down handler
2022-02-01 15:08:29,275 INFO    WriterThread:3248864 [datastore.py:close():281] close: /mnt/lv_ai_1_dante/javierr/wav2vec2-xls-r-1b-npsc/wandb/run-20220201_150818-33gv5m8t/run-33gv5m8t.wandb
2022-02-01 15:08:29,856 INFO    SenderThread:3248864 [sender.py:finish():1070] shutting down sender
2022-02-01 15:08:29,856 INFO    SenderThread:3248864 [file_pusher.py:finish():177] shutting down file pusher
2022-02-01 15:08:29,856 INFO    SenderThread:3248864 [file_pusher.py:join():182] waiting for file pusher
2022-02-01 15:08:29,859 INFO    MainThread:3248864 [internal.py:handle_exit():77] Internal process exited