Hugging Face's logo

Finnish-NLP
/
whisper-large-finnish-v3

Automatic Speech Recognition
Transformers
TensorBoard
Safetensors
Finnish
whisper
whisper-event
finnish
speech-recognition
Eval Results
Inference Endpoints
Model card Files Metrics Community
whisper-large-finnish-v3
File size: 40,634 Bytes 
b0ae254
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
 
2023-11-18 12:43:02,581 INFO    StreamThr :45455 [internal.py:wandb_internal():86] W&B internal server running at pid: 45455, started at: 2023-11-18 12:43:02.579911
2023-11-18 12:43:02,582 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status
2023-11-18 12:43:02,587 INFO    WriterThread:45455 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_124302-mpnwkerp/run-mpnwkerp.wandb
2023-11-18 12:43:02,589 DEBUG   SenderThread:45455 [sender.py:send():380] send: header
2023-11-18 12:43:02,589 DEBUG   SenderThread:45455 [sender.py:send():380] send: run
2023-11-18 12:43:03,267 INFO    SenderThread:45455 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files
2023-11-18 12:43:03,276 INFO    SenderThread:45455 [sender.py:_start_run_threads():1124] run started: mpnwkerp with start time 1700304182.581382
2023-11-18 12:43:03,277 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: check_version
2023-11-18 12:43:03,277 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: check_version
2023-11-18 12:43:03,329 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: run_start
2023-11-18 12:43:03,359 DEBUG   HandlerThread:45455 [system_info.py:__init__():32] System info init
2023-11-18 12:43:03,359 DEBUG   HandlerThread:45455 [system_info.py:__init__():47] System info init done
2023-11-18 12:43:03,360 INFO    HandlerThread:45455 [system_monitor.py:start():194] Starting system monitor
2023-11-18 12:43:03,360 INFO    SystemMonitor:45455 [system_monitor.py:_start():158] Starting system asset monitoring threads
2023-11-18 12:43:03,361 INFO    HandlerThread:45455 [system_monitor.py:probe():214] Collecting system info
2023-11-18 12:43:03,362 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started cpu monitoring
2023-11-18 12:43:03,362 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started disk monitoring
2023-11-18 12:43:03,364 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started gpu monitoring
2023-11-18 12:43:03,365 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started memory monitoring
2023-11-18 12:43:03,366 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started network monitoring
2023-11-18 12:43:03,374 DEBUG   HandlerThread:45455 [system_info.py:probe():196] Probing system
2023-11-18 12:43:03,490 DEBUG   HandlerThread:45455 [gitlib.py:_init_repo():56] git repository is invalid
2023-11-18 12:43:03,490 DEBUG   HandlerThread:45455 [system_info.py:probe():244] Probing system done
2023-11-18 12:43:03,491 DEBUG   HandlerThread:45455 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T10:43:03.374908', 'startedAt': '2023-11-18T10:43:02.543043', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=60000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=2', '--logging_steps=20', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.412132263183594}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}}
2023-11-18 12:43:03,491 INFO    HandlerThread:45455 [system_monitor.py:probe():224] Finished collecting system info
2023-11-18 12:43:03,491 INFO    HandlerThread:45455 [system_monitor.py:probe():227] Publishing system info
2023-11-18 12:43:03,491 DEBUG   HandlerThread:45455 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
2023-11-18 12:43:03,493 DEBUG   HandlerThread:45455 [system_info.py:_save_pip():68] Saving pip packages done
2023-11-18 12:43:03,495 DEBUG   HandlerThread:45455 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
2023-11-18 12:43:04,270 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/conda-environment.yaml
2023-11-18 12:43:04,270 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/requirements.txt
2023-11-18 12:43:06,591 DEBUG   HandlerThread:45455 [system_info.py:_save_conda():87] Saving conda packages done
2023-11-18 12:43:06,593 INFO    HandlerThread:45455 [system_monitor.py:probe():229] Finished publishing system info
2023-11-18 12:43:06,606 DEBUG   SenderThread:45455 [sender.py:send():380] send: files
2023-11-18 12:43:06,607 INFO    SenderThread:45455 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now
2023-11-18 12:43:06,613 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:43:06,614 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:43:06,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:43:06,852 DEBUG   SenderThread:45455 [sender.py:send():380] send: telemetry
2023-11-18 12:43:06,852 DEBUG   SenderThread:45455 [sender.py:send():380] send: config
2023-11-18 12:43:06,853 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:43:06,853 DEBUG   SenderThread:45455 [sender.py:send():380] send: telemetry
2023-11-18 12:43:06,853 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:43:06,854 WARNING SenderThread:45455 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen)
2023-11-18 12:43:06,854 DEBUG   SenderThread:45455 [sender.py:send():380] send: telemetry
2023-11-18 12:43:07,206 INFO    wandb-upload_0:45455 [upload_job.py:push():131] Uploaded file /tmp/tmplz3npzmzwandb/6cmzyt37-wandb-metadata.json
2023-11-18 12:43:07,301 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/conda-environment.yaml
2023-11-18 12:43:07,302 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/wandb-metadata.json
2023-11-18 12:43:07,303 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:43:07,963 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:09,312 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:43:11,321 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:43:13,764 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:18,765 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:21,613 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:43:21,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:43:21,614 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:43:23,781 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:28,782 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:33,787 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:34,428 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/config.yaml
2023-11-18 12:43:36,613 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:43:36,614 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:43:36,655 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:43:39,866 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:44,867 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:49,868 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:51,613 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:43:51,614 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:43:51,655 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:43:55,826 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:00,827 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:03,367 DEBUG   SystemMonitor:45455 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2023-11-18 12:44:03,370 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:44:06,371 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:06,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:44:06,615 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:44:06,655 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:44:11,830 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:16,831 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:21,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:44:21,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:44:21,615 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:44:21,875 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:26,876 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:31,876 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:33,371 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:44:36,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:44:36,615 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:44:36,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:44:37,778 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:42,779 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:47,780 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:51,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:44:51,615 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:44:51,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:44:52,813 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:57,814 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:02,814 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:03,371 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:45:05,778 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:06,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:45:06,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:45:06,616 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:45:07,790 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:07,837 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:12,838 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:18,249 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:19,127 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:21,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:45:21,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:45:21,616 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:45:23,798 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:29,756 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:31,459 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:33,381 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:45:35,382 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:36,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:45:36,668 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:45:36,669 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:45:39,503 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:40,804 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:45,806 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:49,578 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:51,663 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:51,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:45:51,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:45:51,666 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:45:56,871 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:59,640 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:02,660 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:03,372 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:46:06,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:46:06,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:46:06,666 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:46:07,696 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:07,895 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:12,896 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:17,763 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:18,617 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:21,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:46:21,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:46:21,666 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:46:23,904 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:27,817 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:29,138 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:33,373 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:46:34,994 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:35,873 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:36,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:46:36,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:46:36,666 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:46:40,890 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:46,708 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:47,941 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:51,715 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:52,973 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:46:52,974 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:46:52,975 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:46:56,002 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:57,218 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:02,219 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:03,374 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:47:06,051 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:07,402 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:07,970 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:47:07,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:47:07,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:47:12,424 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:14,099 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:17,425 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:22,970 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:47:22,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:47:22,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:47:23,216 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:24,154 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:28,217 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:33,213 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:33,375 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:47:33,375 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:37,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:47:37,973 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:47:37,974 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:47:39,132 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:41,268 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:44,743 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:49,745 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:51,327 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:52,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:47:52,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:47:52,972 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:47:55,178 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:00,879 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:01,381 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:03,376 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:48:06,377 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:07,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:48:07,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:48:07,979 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:48:09,427 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:12,147 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:17,149 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:17,445 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: partial_history
2023-11-18 12:48:17,447 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:48:17,448 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:48:17,448 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:48:17,448 DEBUG   SenderThread:45455 [sender.py:send():380] send: history
2023-11-18 12:48:17,449 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: summary_record
2023-11-18 12:48:17,459 INFO    SenderThread:45455 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 12:48:17,494 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/wandb-summary.json
2023-11-18 12:48:19,508 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:22,474 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:22,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:48:22,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:48:22,979 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:48:23,530 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/config.yaml
2023-11-18 12:48:28,127 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:31,605 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:33,376 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:48:33,377 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:37,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:48:37,972 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:48:37,980 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:48:39,159 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:39,663 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:44,160 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:49,733 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:50,056 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:52,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:48:52,972 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:48:52,980 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:48:55,233 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:57,792 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:00,242 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:03,378 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:49:05,380 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:07,856 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:08,022 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:49:08,022 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:49:08,023 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:49:11,321 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:15,912 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:16,672 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:21,673 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:23,021 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:49:23,022 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:49:23,022 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:49:27,289 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:30,007 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:33,036 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:33,389 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:49:38,021 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:49:38,022 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:49:38,022 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:49:38,274 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:40,064 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:43,276 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:48,114 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:48,816 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:53,818 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:53,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:49:53,939 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:49:53,940 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:49:58,262 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:58,988 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:03,380 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:50:03,988 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:06,318 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:08,936 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:50:08,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:50:08,937 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:50:09,231 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:14,234 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:16,394 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:19,934 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:23,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:50:23,938 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:50:23,938 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:50:25,101 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:25,459 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:30,102 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:33,381 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:50:35,570 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:35,722 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:38,936 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:50:38,937 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:50:38,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:50:41,189 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:45,644 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:46,352 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:51,355 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:53,718 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:53,936 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:50:53,937 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:50:53,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:50:57,135 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:03,005 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:03,382 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:51:03,778 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:51:08,383 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:10,578 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:51:10,579 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:51:10,579 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:51:13,830 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:13,852 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:51:18,831 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:23,928 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:51:24,005 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report