Hugging Face's logo

Finnish-NLP
/
whisper-large-finnish-v3

Automatic Speech Recognition
Transformers
TensorBoard
Safetensors
Finnish
whisper
whisper-event
finnish
speech-recognition
Eval Results
Inference Endpoints
Model card Files Metrics Community
1
whisper-large-finnish-v3
File size: 31,223 Bytes 
b0ae254
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
 
2023-11-18 11:42:08,966 INFO    StreamThr :13645 [internal.py:wandb_internal():86] W&B internal server running at pid: 13645, started at: 2023-11-18 11:42:08.962667
2023-11-18 11:42:08,970 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status
2023-11-18 11:42:08,972 INFO    WriterThread:13645 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_114208-iofrt503/run-iofrt503.wandb
2023-11-18 11:42:08,977 DEBUG   SenderThread:13645 [sender.py:send():380] send: header
2023-11-18 11:42:08,978 DEBUG   SenderThread:13645 [sender.py:send():380] send: run
2023-11-18 11:42:09,383 INFO    SenderThread:13645 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_114208-iofrt503/files
2023-11-18 11:42:09,392 INFO    SenderThread:13645 [sender.py:_start_run_threads():1124] run started: iofrt503 with start time 1700300528.952379
2023-11-18 11:42:09,393 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: check_version
2023-11-18 11:42:09,394 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: check_version
2023-11-18 11:42:09,547 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: run_start
2023-11-18 11:42:09,556 DEBUG   HandlerThread:13645 [system_info.py:__init__():32] System info init
2023-11-18 11:42:09,556 DEBUG   HandlerThread:13645 [system_info.py:__init__():47] System info init done
2023-11-18 11:42:09,556 INFO    HandlerThread:13645 [system_monitor.py:start():194] Starting system monitor
2023-11-18 11:42:09,557 INFO    SystemMonitor:13645 [system_monitor.py:_start():158] Starting system asset monitoring threads
2023-11-18 11:42:09,557 INFO    HandlerThread:13645 [system_monitor.py:probe():214] Collecting system info
2023-11-18 11:42:09,558 INFO    SystemMonitor:13645 [interfaces.py:start():190] Started cpu monitoring
2023-11-18 11:42:09,559 INFO    SystemMonitor:13645 [interfaces.py:start():190] Started disk monitoring
2023-11-18 11:42:09,561 INFO    SystemMonitor:13645 [interfaces.py:start():190] Started gpu monitoring
2023-11-18 11:42:09,562 INFO    SystemMonitor:13645 [interfaces.py:start():190] Started memory monitoring
2023-11-18 11:42:09,562 INFO    SystemMonitor:13645 [interfaces.py:start():190] Started network monitoring
2023-11-18 11:42:09,573 DEBUG   HandlerThread:13645 [system_info.py:probe():196] Probing system
2023-11-18 11:42:09,658 DEBUG   HandlerThread:13645 [gitlib.py:_init_repo():56] git repository is invalid
2023-11-18 11:42:09,658 DEBUG   HandlerThread:13645 [system_info.py:probe():244] Probing system done
2023-11-18 11:42:09,659 DEBUG   HandlerThread:13645 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T09:42:09.573567', 'startedAt': '2023-11-18T09:42:08.901047', 'docker': None, 'cuda': None, 'args': ('--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v2', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=Whisper Large V3 Finnish', '--max_steps=30000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=4', '--logging_steps=25', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.273990631103516}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}}
2023-11-18 11:42:09,659 INFO    HandlerThread:13645 [system_monitor.py:probe():224] Finished collecting system info
2023-11-18 11:42:09,659 INFO    HandlerThread:13645 [system_monitor.py:probe():227] Publishing system info
2023-11-18 11:42:09,659 DEBUG   HandlerThread:13645 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
2023-11-18 11:42:09,662 DEBUG   HandlerThread:13645 [system_info.py:_save_pip():68] Saving pip packages done
2023-11-18 11:42:09,662 DEBUG   HandlerThread:13645 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
2023-11-18 11:42:10,387 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/requirements.txt
2023-11-18 11:42:10,388 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/conda-environment.yaml
2023-11-18 11:42:11,825 DEBUG   HandlerThread:13645 [system_info.py:_save_conda():87] Saving conda packages done
2023-11-18 11:42:11,829 INFO    HandlerThread:13645 [system_monitor.py:probe():229] Finished publishing system info
2023-11-18 11:42:11,832 DEBUG   SenderThread:13645 [sender.py:send():380] send: files
2023-11-18 11:42:11,832 INFO    SenderThread:13645 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now
2023-11-18 11:42:11,841 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:42:11,841 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:42:11,842 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:42:12,031 DEBUG   SenderThread:13645 [sender.py:send():380] send: telemetry
2023-11-18 11:42:12,031 DEBUG   SenderThread:13645 [sender.py:send():380] send: config
2023-11-18 11:42:12,032 DEBUG   SenderThread:13645 [sender.py:send():380] send: metric
2023-11-18 11:42:12,032 DEBUG   SenderThread:13645 [sender.py:send():380] send: telemetry
2023-11-18 11:42:12,032 DEBUG   SenderThread:13645 [sender.py:send():380] send: metric
2023-11-18 11:42:12,033 WARNING SenderThread:13645 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen)
2023-11-18 11:42:12,033 DEBUG   SenderThread:13645 [sender.py:send():380] send: telemetry
2023-11-18 11:42:12,399 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/conda-environment.yaml
2023-11-18 11:42:12,400 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-metadata.json
2023-11-18 11:42:12,402 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log
2023-11-18 11:42:12,438 INFO    wandb-upload_0:13645 [upload_job.py:push():131] Uploaded file /tmp/tmpcostdywiwandb/1tczprur-wandb-metadata.json
2023-11-18 11:42:14,150 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:42:14,410 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log
2023-11-18 11:42:16,429 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log
2023-11-18 11:42:19,311 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:42:24,312 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:42:26,840 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:42:26,840 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:42:26,842 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:42:30,022 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:42:35,023 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:42:40,027 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:42:40,566 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/config.yaml
2023-11-18 11:42:41,840 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:42:41,841 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:42:41,885 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:42:46,043 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:42:51,045 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:42:56,046 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:42:56,840 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:42:56,841 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:42:56,885 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:43:01,067 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:06,068 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:09,563 DEBUG   SystemMonitor:13645 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2023-11-18 11:43:09,565 DEBUG   SenderThread:13645 [sender.py:send():380] send: stats
2023-11-18 11:43:11,566 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:11,840 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:43:11,841 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:43:11,885 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:43:17,026 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:22,027 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:26,841 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:43:26,842 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:43:26,885 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:43:27,159 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:32,160 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:37,161 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:39,566 DEBUG   SenderThread:13645 [sender.py:send():380] send: stats
2023-11-18 11:43:41,840 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:43:41,841 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:43:41,885 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:43:43,059 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:48,070 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:53,071 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:43:56,841 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:43:56,842 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:43:56,885 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:43:58,105 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:44:03,106 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:44:08,107 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:44:09,566 DEBUG   SenderThread:13645 [sender.py:send():380] send: stats
2023-11-18 11:44:11,841 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:44:11,841 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:44:11,842 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:44:13,023 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log
2023-11-18 11:44:13,119 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:44:17,048 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log
2023-11-18 11:44:17,519 DEBUG   SenderThread:13645 [sender.py:send():380] send: exit
2023-11-18 11:44:17,520 INFO    SenderThread:13645 [sender.py:send_exit():585] handling exit code: 1
2023-11-18 11:44:17,520 INFO    SenderThread:13645 [sender.py:send_exit():587] handling runtime: 127
2023-11-18 11:44:17,531 INFO    SenderThread:13645 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 11:44:17,532 INFO    SenderThread:13645 [sender.py:send_exit():593] send defer
2023-11-18 11:44:17,532 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:17,532 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 0
2023-11-18 11:44:17,532 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:17,533 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 0
2023-11-18 11:44:17,533 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 1
2023-11-18 11:44:17,533 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:17,533 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 1
2023-11-18 11:44:17,534 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:17,534 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 1
2023-11-18 11:44:17,534 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 2
2023-11-18 11:44:17,534 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:17,535 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 2
2023-11-18 11:44:17,535 INFO    HandlerThread:13645 [system_monitor.py:finish():203] Stopping system monitor
2023-11-18 11:44:17,535 DEBUG   SystemMonitor:13645 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2023-11-18 11:44:17,536 DEBUG   SystemMonitor:13645 [system_monitor.py:_start():183] Publishing last batch of metrics
2023-11-18 11:44:17,536 INFO    HandlerThread:13645 [interfaces.py:finish():202] Joined cpu monitor
2023-11-18 11:44:17,537 INFO    HandlerThread:13645 [interfaces.py:finish():202] Joined disk monitor
2023-11-18 11:44:17,548 INFO    HandlerThread:13645 [interfaces.py:finish():202] Joined gpu monitor
2023-11-18 11:44:17,549 INFO    HandlerThread:13645 [interfaces.py:finish():202] Joined memory monitor
2023-11-18 11:44:17,549 INFO    HandlerThread:13645 [interfaces.py:finish():202] Joined network monitor
2023-11-18 11:44:17,550 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:17,550 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 2
2023-11-18 11:44:17,550 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 3
2023-11-18 11:44:17,550 DEBUG   SenderThread:13645 [sender.py:send():380] send: stats
2023-11-18 11:44:17,551 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:17,551 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 3
2023-11-18 11:44:17,551 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:17,552 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 3
2023-11-18 11:44:17,552 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 4
2023-11-18 11:44:17,552 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:17,552 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 4
2023-11-18 11:44:17,553 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:17,553 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 4
2023-11-18 11:44:17,553 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 5
2023-11-18 11:44:17,553 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:17,553 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 5
2023-11-18 11:44:17,554 DEBUG   SenderThread:13645 [sender.py:send():380] send: summary
2023-11-18 11:44:17,556 INFO    SenderThread:13645 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 11:44:17,557 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:17,557 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 5
2023-11-18 11:44:17,557 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 6
2023-11-18 11:44:17,557 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:17,557 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 6
2023-11-18 11:44:17,558 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:17,558 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 6
2023-11-18 11:44:17,558 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 7
2023-11-18 11:44:17,558 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:44:17,559 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:17,559 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 7
2023-11-18 11:44:17,559 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:17,559 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 7
2023-11-18 11:44:18,055 INFO    Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-summary.json
2023-11-18 11:44:18,183 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 8
2023-11-18 11:44:18,184 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:18,184 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 8
2023-11-18 11:44:18,184 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:18,185 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 8
2023-11-18 11:44:18,185 INFO    SenderThread:13645 [job_builder.py:build():298] Attempting to build job artifact
2023-11-18 11:44:18,191 INFO    SenderThread:13645 [job_builder.py:_get_source_type():439] no source found
2023-11-18 11:44:18,191 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 9
2023-11-18 11:44:18,192 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:18,192 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 9
2023-11-18 11:44:18,192 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:18,193 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 9
2023-11-18 11:44:18,193 INFO    SenderThread:13645 [dir_watcher.py:finish():358] shutting down directory watcher
2023-11-18 11:44:18,519 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:44:19,072 INFO    SenderThread:13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log
2023-11-18 11:44:19,073 INFO    SenderThread:13645 [dir_watcher.py:finish():388] scan: /mnt/e/wandb/run-20231118_114208-iofrt503/files
2023-11-18 11:44:19,075 INFO    SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/conda-environment.yaml conda-environment.yaml
2023-11-18 11:44:19,078 INFO    SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/config.yaml config.yaml
2023-11-18 11:44:19,086 INFO    SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log output.log
2023-11-18 11:44:19,092 INFO    SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/requirements.txt requirements.txt
2023-11-18 11:44:19,098 INFO    SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-metadata.json wandb-metadata.json
2023-11-18 11:44:19,098 INFO    SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-summary.json wandb-summary.json
2023-11-18 11:44:19,103 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 10
2023-11-18 11:44:19,104 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:44:19,104 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:19,105 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 10
2023-11-18 11:44:19,108 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:19,108 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 10
2023-11-18 11:44:19,108 INFO    SenderThread:13645 [file_pusher.py:finish():175] shutting down file pusher
2023-11-18 11:44:19,496 INFO    wandb-upload_0:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/conda-environment.yaml
2023-11-18 11:44:19,520 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:44:19,521 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:44:19,572 INFO    wandb-upload_1:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/config.yaml
2023-11-18 11:44:19,612 INFO    wandb-upload_4:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-summary.json
2023-11-18 11:44:19,633 INFO    wandb-upload_2:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log
2023-11-18 11:44:19,642 INFO    wandb-upload_3:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/requirements.txt
2023-11-18 11:44:19,843 INFO    Thread-11 (_thread_body):13645 [sender.py:transition_state():613] send defer: 11
2023-11-18 11:44:19,844 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:19,844 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 11
2023-11-18 11:44:19,845 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:19,845 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 11
2023-11-18 11:44:19,845 INFO    SenderThread:13645 [file_pusher.py:join():181] waiting for file pusher
2023-11-18 11:44:19,846 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 12
2023-11-18 11:44:19,846 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:19,846 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 12
2023-11-18 11:44:19,847 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:19,847 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 12
2023-11-18 11:44:19,847 INFO    SenderThread:13645 [file_stream.py:finish():595] file stream finish called
2023-11-18 11:44:20,197 INFO    SenderThread:13645 [file_stream.py:finish():599] file stream finish is done
2023-11-18 11:44:20,197 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 13
2023-11-18 11:44:20,198 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:20,198 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 13
2023-11-18 11:44:20,199 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:20,199 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 13
2023-11-18 11:44:20,199 INFO    SenderThread:13645 [sender.py:transition_state():613] send defer: 14
2023-11-18 11:44:20,200 DEBUG   SenderThread:13645 [sender.py:send():380] send: final
2023-11-18 11:44:20,200 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:44:20,200 DEBUG   SenderThread:13645 [sender.py:send():380] send: footer
2023-11-18 11:44:20,201 INFO    HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 14
2023-11-18 11:44:20,202 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: defer
2023-11-18 11:44:20,202 INFO    SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 14
2023-11-18 11:44:20,203 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:44:20,203 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:44:20,204 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:44:20,204 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: server_info
2023-11-18 11:44:20,204 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:44:20,205 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: get_summary
2023-11-18 11:44:20,205 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: server_info
2023-11-18 11:44:20,206 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: sampled_history
2023-11-18 11:44:20,208 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:44:20,209 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: job_info
2023-11-18 11:44:20,371 DEBUG   SenderThread:13645 [sender.py:send_request():407] send_request: job_info
2023-11-18 11:44:20,371 INFO    MainThread:13645 [wandb_run.py:_footer_history_summary_info():3760] rendering history
2023-11-18 11:44:20,372 INFO    MainThread:13645 [wandb_run.py:_footer_history_summary_info():3792] rendering summary
2023-11-18 11:44:20,372 INFO    MainThread:13645 [wandb_run.py:_footer_sync_info():3719] logging synced files
2023-11-18 11:44:20,372 DEBUG   HandlerThread:13645 [handler.py:handle_request():146] handle_request: shutdown
2023-11-18 11:44:20,372 INFO    HandlerThread:13645 [handler.py:finish():866] shutting down handler
2023-11-18 11:44:21,209 INFO    WriterThread:13645 [datastore.py:close():294] close: /mnt/e/wandb/run-20231118_114208-iofrt503/run-iofrt503.wandb
2023-11-18 11:44:21,371 INFO    SenderThread:13645 [sender.py:finish():1536] shutting down sender
2023-11-18 11:44:21,372 INFO    SenderThread:13645 [file_pusher.py:finish():175] shutting down file pusher
2023-11-18 11:44:21,372 INFO    SenderThread:13645 [file_pusher.py:join():181] waiting for file pusher