Hugging Face's logo

Finnish-NLP
/
whisper-large-finnish-v3

Automatic Speech Recognition
Transformers
TensorBoard
Safetensors
Finnish
whisper
whisper-event
finnish
speech-recognition
Eval Results
Inference Endpoints
Model card Files Metrics Community
1
whisper-large-finnish-v3
File size: 30,765 Bytes 
b0ae254
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
 
2023-11-18 11:46:07,262 INFO    StreamThr :15138 [internal.py:wandb_internal():86] W&B internal server running at pid: 15138, started at: 2023-11-18 11:46:07.260162
2023-11-18 11:46:07,264 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status
2023-11-18 11:46:07,268 INFO    WriterThread:15138 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_114607-nijw2jcj/run-nijw2jcj.wandb
2023-11-18 11:46:07,270 DEBUG   SenderThread:15138 [sender.py:send():380] send: header
2023-11-18 11:46:07,271 DEBUG   SenderThread:15138 [sender.py:send():380] send: run
2023-11-18 11:46:07,725 INFO    SenderThread:15138 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files
2023-11-18 11:46:07,734 INFO    SenderThread:15138 [sender.py:_start_run_threads():1124] run started: nijw2jcj with start time 1700300767.260905
2023-11-18 11:46:07,739 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: check_version
2023-11-18 11:46:07,739 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: check_version
2023-11-18 11:46:07,788 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: run_start
2023-11-18 11:46:07,795 DEBUG   HandlerThread:15138 [system_info.py:__init__():32] System info init
2023-11-18 11:46:07,795 DEBUG   HandlerThread:15138 [system_info.py:__init__():47] System info init done
2023-11-18 11:46:07,795 INFO    HandlerThread:15138 [system_monitor.py:start():194] Starting system monitor
2023-11-18 11:46:07,796 INFO    SystemMonitor:15138 [system_monitor.py:_start():158] Starting system asset monitoring threads
2023-11-18 11:46:07,796 INFO    HandlerThread:15138 [system_monitor.py:probe():214] Collecting system info
2023-11-18 11:46:07,797 INFO    SystemMonitor:15138 [interfaces.py:start():190] Started cpu monitoring
2023-11-18 11:46:07,799 INFO    SystemMonitor:15138 [interfaces.py:start():190] Started disk monitoring
2023-11-18 11:46:07,800 INFO    SystemMonitor:15138 [interfaces.py:start():190] Started gpu monitoring
2023-11-18 11:46:07,801 INFO    SystemMonitor:15138 [interfaces.py:start():190] Started memory monitoring
2023-11-18 11:46:07,802 INFO    SystemMonitor:15138 [interfaces.py:start():190] Started network monitoring
2023-11-18 11:46:07,807 DEBUG   HandlerThread:15138 [system_info.py:probe():196] Probing system
2023-11-18 11:46:07,927 DEBUG   HandlerThread:15138 [gitlib.py:_init_repo():56] git repository is invalid
2023-11-18 11:46:07,927 DEBUG   HandlerThread:15138 [system_info.py:probe():244] Probing system done
2023-11-18 11:46:07,928 DEBUG   HandlerThread:15138 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T09:46:07.807399', 'startedAt': '2023-11-18T09:46:07.220916', 'docker': None, 'cuda': None, 'args': ('--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v2', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=Whisper Large V3 Finnish', '--max_steps=30000', '--output_dir=./', '--per_device_train_batch_size=4', '--per_device_eval_batch_size=2', '--gradient_accumulation_steps=4', '--logging_steps=25', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.274234771728516}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}}
2023-11-18 11:46:07,928 INFO    HandlerThread:15138 [system_monitor.py:probe():224] Finished collecting system info
2023-11-18 11:46:07,928 INFO    HandlerThread:15138 [system_monitor.py:probe():227] Publishing system info
2023-11-18 11:46:07,928 DEBUG   HandlerThread:15138 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
2023-11-18 11:46:07,930 DEBUG   HandlerThread:15138 [system_info.py:_save_pip():68] Saving pip packages done
2023-11-18 11:46:07,931 DEBUG   HandlerThread:15138 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
2023-11-18 11:46:08,731 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/requirements.txt
2023-11-18 11:46:08,734 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/conda-environment.yaml
2023-11-18 11:46:10,342 DEBUG   HandlerThread:15138 [system_info.py:_save_conda():87] Saving conda packages done
2023-11-18 11:46:10,345 INFO    HandlerThread:15138 [system_monitor.py:probe():229] Finished publishing system info
2023-11-18 11:46:10,347 DEBUG   SenderThread:15138 [sender.py:send():380] send: files
2023-11-18 11:46:10,348 INFO    SenderThread:15138 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now
2023-11-18 11:46:10,357 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:46:10,358 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:46:10,358 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:46:10,585 DEBUG   SenderThread:15138 [sender.py:send():380] send: telemetry
2023-11-18 11:46:10,585 DEBUG   SenderThread:15138 [sender.py:send():380] send: config
2023-11-18 11:46:10,586 DEBUG   SenderThread:15138 [sender.py:send():380] send: metric
2023-11-18 11:46:10,586 DEBUG   SenderThread:15138 [sender.py:send():380] send: telemetry
2023-11-18 11:46:10,586 DEBUG   SenderThread:15138 [sender.py:send():380] send: metric
2023-11-18 11:46:10,586 WARNING SenderThread:15138 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen)
2023-11-18 11:46:10,587 DEBUG   SenderThread:15138 [sender.py:send():380] send: telemetry
2023-11-18 11:46:10,738 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/conda-environment.yaml
2023-11-18 11:46:10,739 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log
2023-11-18 11:46:10,739 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-metadata.json
2023-11-18 11:46:12,338 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:46:12,731 INFO    wandb-upload_0:15138 [upload_job.py:push():131] Uploaded file /tmp/tmpa4run1f0wandb/ypjb894q-wandb-metadata.json
2023-11-18 11:46:12,746 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log
2023-11-18 11:46:14,754 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log
2023-11-18 11:46:17,498 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:46:22,499 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:46:25,356 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:46:25,357 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:46:25,357 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:46:27,587 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:46:32,588 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:46:37,592 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:46:37,851 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/config.yaml
2023-11-18 11:46:40,357 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:46:40,357 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:46:40,405 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:46:43,525 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:46:48,525 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:46:53,526 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:46:55,357 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:46:55,357 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:46:55,405 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:46:58,604 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:03,604 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:07,802 DEBUG   SystemMonitor:15138 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2023-11-18 11:47:07,803 DEBUG   SenderThread:15138 [sender.py:send():380] send: stats
2023-11-18 11:47:08,804 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:10,357 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:47:10,357 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:47:10,405 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:47:14,615 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:19,615 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:24,616 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:25,357 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:47:25,358 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:47:25,405 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:47:30,557 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:35,558 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:37,804 DEBUG   SenderThread:15138 [sender.py:send():380] send: stats
2023-11-18 11:47:40,357 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:47:40,358 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:47:40,405 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:47:41,534 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:46,535 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:51,536 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:47:55,358 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:47:55,358 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:47:55,405 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:47:56,604 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:48:01,605 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:48:06,654 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:48:07,194 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log
2023-11-18 11:48:07,805 DEBUG   SenderThread:15138 [sender.py:send():380] send: stats
2023-11-18 11:48:09,201 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log
2023-11-18 11:48:10,113 DEBUG   SenderThread:15138 [sender.py:send():380] send: exit
2023-11-18 11:48:10,113 INFO    SenderThread:15138 [sender.py:send_exit():585] handling exit code: 1
2023-11-18 11:48:10,113 INFO    SenderThread:15138 [sender.py:send_exit():587] handling runtime: 122
2023-11-18 11:48:10,125 INFO    SenderThread:15138 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 11:48:10,127 INFO    SenderThread:15138 [sender.py:send_exit():593] send defer
2023-11-18 11:48:10,127 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,127 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 0
2023-11-18 11:48:10,128 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,128 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 0
2023-11-18 11:48:10,128 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 1
2023-11-18 11:48:10,128 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,128 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 1
2023-11-18 11:48:10,129 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,129 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 1
2023-11-18 11:48:10,129 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 2
2023-11-18 11:48:10,129 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,130 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 2
2023-11-18 11:48:10,130 INFO    HandlerThread:15138 [system_monitor.py:finish():203] Stopping system monitor
2023-11-18 11:48:10,130 DEBUG   SystemMonitor:15138 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2023-11-18 11:48:10,130 DEBUG   SystemMonitor:15138 [system_monitor.py:_start():183] Publishing last batch of metrics
2023-11-18 11:48:10,131 INFO    HandlerThread:15138 [interfaces.py:finish():202] Joined cpu monitor
2023-11-18 11:48:10,131 INFO    HandlerThread:15138 [interfaces.py:finish():202] Joined disk monitor
2023-11-18 11:48:10,138 INFO    HandlerThread:15138 [interfaces.py:finish():202] Joined gpu monitor
2023-11-18 11:48:10,138 INFO    HandlerThread:15138 [interfaces.py:finish():202] Joined memory monitor
2023-11-18 11:48:10,138 INFO    HandlerThread:15138 [interfaces.py:finish():202] Joined network monitor
2023-11-18 11:48:10,139 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,139 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 2
2023-11-18 11:48:10,139 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 3
2023-11-18 11:48:10,140 DEBUG   SenderThread:15138 [sender.py:send():380] send: stats
2023-11-18 11:48:10,140 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,140 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 3
2023-11-18 11:48:10,141 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,141 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 3
2023-11-18 11:48:10,141 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 4
2023-11-18 11:48:10,141 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,141 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 4
2023-11-18 11:48:10,141 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,142 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 4
2023-11-18 11:48:10,142 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 5
2023-11-18 11:48:10,142 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,142 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 5
2023-11-18 11:48:10,143 DEBUG   SenderThread:15138 [sender.py:send():380] send: summary
2023-11-18 11:48:10,144 INFO    SenderThread:15138 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 11:48:10,145 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,145 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 5
2023-11-18 11:48:10,145 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 6
2023-11-18 11:48:10,146 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,146 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 6
2023-11-18 11:48:10,146 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,146 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 6
2023-11-18 11:48:10,146 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 7
2023-11-18 11:48:10,146 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:48:10,147 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,147 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 7
2023-11-18 11:48:10,147 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,147 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 7
2023-11-18 11:48:10,205 INFO    Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-summary.json
2023-11-18 11:48:10,727 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 8
2023-11-18 11:48:10,727 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,728 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 8
2023-11-18 11:48:10,728 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,729 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 8
2023-11-18 11:48:10,729 INFO    SenderThread:15138 [job_builder.py:build():298] Attempting to build job artifact
2023-11-18 11:48:10,733 INFO    SenderThread:15138 [job_builder.py:_get_source_type():439] no source found
2023-11-18 11:48:10,734 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 9
2023-11-18 11:48:10,734 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:10,734 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 9
2023-11-18 11:48:10,735 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:10,735 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 9
2023-11-18 11:48:10,735 INFO    SenderThread:15138 [dir_watcher.py:finish():358] shutting down directory watcher
2023-11-18 11:48:11,113 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:48:11,219 INFO    SenderThread:15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log
2023-11-18 11:48:11,220 INFO    SenderThread:15138 [dir_watcher.py:finish():388] scan: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files
2023-11-18 11:48:11,222 INFO    SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/conda-environment.yaml conda-environment.yaml
2023-11-18 11:48:11,225 INFO    SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/config.yaml config.yaml
2023-11-18 11:48:11,229 INFO    SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log output.log
2023-11-18 11:48:11,233 INFO    SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/requirements.txt requirements.txt
2023-11-18 11:48:11,237 INFO    SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-metadata.json wandb-metadata.json
2023-11-18 11:48:11,238 INFO    SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-summary.json wandb-summary.json
2023-11-18 11:48:11,241 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 10
2023-11-18 11:48:11,242 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:48:11,242 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:11,242 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 10
2023-11-18 11:48:11,243 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:11,244 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 10
2023-11-18 11:48:11,244 INFO    SenderThread:15138 [file_pusher.py:finish():175] shutting down file pusher
2023-11-18 11:48:11,604 INFO    wandb-upload_0:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/conda-environment.yaml
2023-11-18 11:48:11,638 INFO    wandb-upload_1:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/config.yaml
2023-11-18 11:48:11,717 INFO    wandb-upload_3:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/requirements.txt
2023-11-18 11:48:11,718 INFO    wandb-upload_4:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-summary.json
2023-11-18 11:48:11,733 INFO    wandb-upload_2:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log
2023-11-18 11:48:11,934 INFO    Thread-11 (_thread_body):15138 [sender.py:transition_state():613] send defer: 11
2023-11-18 11:48:11,935 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:11,935 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 11
2023-11-18 11:48:11,936 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:11,936 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 11
2023-11-18 11:48:11,936 INFO    SenderThread:15138 [file_pusher.py:join():181] waiting for file pusher
2023-11-18 11:48:11,937 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 12
2023-11-18 11:48:11,937 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:11,937 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 12
2023-11-18 11:48:11,938 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:11,938 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 12
2023-11-18 11:48:11,939 INFO    SenderThread:15138 [file_stream.py:finish():595] file stream finish called
2023-11-18 11:48:12,113 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:48:12,475 INFO    SenderThread:15138 [file_stream.py:finish():599] file stream finish is done
2023-11-18 11:48:12,475 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 13
2023-11-18 11:48:12,476 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:48:12,476 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:12,476 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 13
2023-11-18 11:48:12,476 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:12,477 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 13
2023-11-18 11:48:12,477 INFO    SenderThread:15138 [sender.py:transition_state():613] send defer: 14
2023-11-18 11:48:12,477 DEBUG   SenderThread:15138 [sender.py:send():380] send: final
2023-11-18 11:48:12,477 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:48:12,477 DEBUG   SenderThread:15138 [sender.py:send():380] send: footer
2023-11-18 11:48:12,478 INFO    HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 14
2023-11-18 11:48:12,478 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: defer
2023-11-18 11:48:12,478 INFO    SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 14
2023-11-18 11:48:12,479 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:48:12,479 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:48:12,479 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:48:12,479 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: server_info
2023-11-18 11:48:12,480 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:48:12,480 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: get_summary
2023-11-18 11:48:12,480 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: server_info
2023-11-18 11:48:12,480 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: sampled_history
2023-11-18 11:48:12,482 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:48:12,482 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: job_info
2023-11-18 11:48:12,642 DEBUG   SenderThread:15138 [sender.py:send_request():407] send_request: job_info
2023-11-18 11:48:12,642 INFO    MainThread:15138 [wandb_run.py:_footer_history_summary_info():3760] rendering history
2023-11-18 11:48:12,642 INFO    MainThread:15138 [wandb_run.py:_footer_history_summary_info():3792] rendering summary
2023-11-18 11:48:12,643 INFO    MainThread:15138 [wandb_run.py:_footer_sync_info():3719] logging synced files
2023-11-18 11:48:12,643 DEBUG   HandlerThread:15138 [handler.py:handle_request():146] handle_request: shutdown
2023-11-18 11:48:12,643 INFO    HandlerThread:15138 [handler.py:finish():866] shutting down handler
2023-11-18 11:48:13,482 INFO    WriterThread:15138 [datastore.py:close():294] close: /mnt/e/wandb/run-20231118_114607-nijw2jcj/run-nijw2jcj.wandb
2023-11-18 11:48:13,642 INFO    SenderThread:15138 [sender.py:finish():1536] shutting down sender
2023-11-18 11:48:13,643 INFO    SenderThread:15138 [file_pusher.py:finish():175] shutting down file pusher
2023-11-18 11:48:13,643 INFO    SenderThread:15138 [file_pusher.py:join():181] waiting for file pusher