2023-11-18 12:43:02,581 INFO    StreamThr :45455 [internal.py:wandb_internal():86] W&B internal server running at pid: 45455, started at: 2023-11-18 12:43:02.579911
2023-11-18 12:43:02,582 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status
2023-11-18 12:43:02,587 INFO    WriterThread:45455 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_124302-mpnwkerp/run-mpnwkerp.wandb
2023-11-18 12:43:02,589 DEBUG   SenderThread:45455 [sender.py:send():380] send: header
2023-11-18 12:43:02,589 DEBUG   SenderThread:45455 [sender.py:send():380] send: run
2023-11-18 12:43:03,267 INFO    SenderThread:45455 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files
2023-11-18 12:43:03,276 INFO    SenderThread:45455 [sender.py:_start_run_threads():1124] run started: mpnwkerp with start time 1700304182.581382
2023-11-18 12:43:03,277 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: check_version
2023-11-18 12:43:03,277 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: check_version
2023-11-18 12:43:03,329 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: run_start
2023-11-18 12:43:03,359 DEBUG   HandlerThread:45455 [system_info.py:__init__():32] System info init
2023-11-18 12:43:03,359 DEBUG   HandlerThread:45455 [system_info.py:__init__():47] System info init done
2023-11-18 12:43:03,360 INFO    HandlerThread:45455 [system_monitor.py:start():194] Starting system monitor
2023-11-18 12:43:03,360 INFO    SystemMonitor:45455 [system_monitor.py:_start():158] Starting system asset monitoring threads
2023-11-18 12:43:03,361 INFO    HandlerThread:45455 [system_monitor.py:probe():214] Collecting system info
2023-11-18 12:43:03,362 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started cpu monitoring
2023-11-18 12:43:03,362 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started disk monitoring
2023-11-18 12:43:03,364 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started gpu monitoring
2023-11-18 12:43:03,365 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started memory monitoring
2023-11-18 12:43:03,366 INFO    SystemMonitor:45455 [interfaces.py:start():190] Started network monitoring
2023-11-18 12:43:03,374 DEBUG   HandlerThread:45455 [system_info.py:probe():196] Probing system
2023-11-18 12:43:03,490 DEBUG   HandlerThread:45455 [gitlib.py:_init_repo():56] git repository is invalid
2023-11-18 12:43:03,490 DEBUG   HandlerThread:45455 [system_info.py:probe():244] Probing system done
2023-11-18 12:43:03,491 DEBUG   HandlerThread:45455 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T10:43:03.374908', 'startedAt': '2023-11-18T10:43:02.543043', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=60000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=2', '--logging_steps=20', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.412132263183594}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}}
2023-11-18 12:43:03,491 INFO    HandlerThread:45455 [system_monitor.py:probe():224] Finished collecting system info
2023-11-18 12:43:03,491 INFO    HandlerThread:45455 [system_monitor.py:probe():227] Publishing system info
2023-11-18 12:43:03,491 DEBUG   HandlerThread:45455 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
2023-11-18 12:43:03,493 DEBUG   HandlerThread:45455 [system_info.py:_save_pip():68] Saving pip packages done
2023-11-18 12:43:03,495 DEBUG   HandlerThread:45455 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
2023-11-18 12:43:04,270 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/conda-environment.yaml
2023-11-18 12:43:04,270 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/requirements.txt
2023-11-18 12:43:06,591 DEBUG   HandlerThread:45455 [system_info.py:_save_conda():87] Saving conda packages done
2023-11-18 12:43:06,593 INFO    HandlerThread:45455 [system_monitor.py:probe():229] Finished publishing system info
2023-11-18 12:43:06,606 DEBUG   SenderThread:45455 [sender.py:send():380] send: files
2023-11-18 12:43:06,607 INFO    SenderThread:45455 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now
2023-11-18 12:43:06,613 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:43:06,614 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:43:06,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:43:06,852 DEBUG   SenderThread:45455 [sender.py:send():380] send: telemetry
2023-11-18 12:43:06,852 DEBUG   SenderThread:45455 [sender.py:send():380] send: config
2023-11-18 12:43:06,853 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:43:06,853 DEBUG   SenderThread:45455 [sender.py:send():380] send: telemetry
2023-11-18 12:43:06,853 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:43:06,854 WARNING SenderThread:45455 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen)
2023-11-18 12:43:06,854 DEBUG   SenderThread:45455 [sender.py:send():380] send: telemetry
2023-11-18 12:43:07,206 INFO    wandb-upload_0:45455 [upload_job.py:push():131] Uploaded file /tmp/tmplz3npzmzwandb/6cmzyt37-wandb-metadata.json
2023-11-18 12:43:07,301 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/conda-environment.yaml
2023-11-18 12:43:07,302 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/wandb-metadata.json
2023-11-18 12:43:07,303 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:43:07,963 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:09,312 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:43:11,321 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:43:13,764 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:18,765 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:21,613 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:43:21,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:43:21,614 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:43:23,781 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:28,782 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:33,787 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:34,428 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/config.yaml
2023-11-18 12:43:36,613 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:43:36,614 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:43:36,655 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:43:39,866 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:44,867 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:49,868 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:43:51,613 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:43:51,614 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:43:51,655 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:43:55,826 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:00,827 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:03,367 DEBUG   SystemMonitor:45455 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2023-11-18 12:44:03,370 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:44:06,371 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:06,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:44:06,615 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:44:06,655 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:44:11,830 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:16,831 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:21,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:44:21,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:44:21,615 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:44:21,875 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:26,876 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:31,876 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:33,371 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:44:36,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:44:36,615 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:44:36,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:44:37,778 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:42,779 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:47,780 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:51,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:44:51,615 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:44:51,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:44:52,813 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:44:57,814 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:02,814 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:03,371 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:45:05,778 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:06,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:45:06,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:45:06,616 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:45:07,790 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:07,837 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:12,838 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:18,249 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:19,127 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:21,614 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:45:21,615 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:45:21,616 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:45:23,798 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:29,756 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:31,459 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:33,381 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:45:35,382 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:36,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:45:36,668 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:45:36,669 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:45:39,503 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:40,804 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:45,806 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:49,578 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:45:51,663 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:51,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:45:51,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:45:51,666 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:45:56,871 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:45:59,640 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:02,660 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:03,372 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:46:06,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:46:06,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:46:06,666 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:46:07,696 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:07,895 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:12,896 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:17,763 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:18,617 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:21,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:46:21,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:46:21,666 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:46:23,904 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:27,817 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:29,138 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:33,373 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:46:34,994 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:35,873 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:36,665 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:46:36,666 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:46:36,666 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:46:40,890 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:46,708 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:47,941 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:51,715 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:46:52,973 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:46:52,974 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:46:52,975 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:46:56,002 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:46:57,218 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:02,219 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:03,374 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:47:06,051 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:07,402 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:07,970 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:47:07,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:47:07,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:47:12,424 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:14,099 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:17,425 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:22,970 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:47:22,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:47:22,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:47:23,216 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:24,154 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:28,217 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:33,213 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:33,375 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:47:33,375 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:37,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:47:37,973 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:47:37,974 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:47:39,132 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:41,268 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:44,743 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:49,745 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:47:51,327 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:47:52,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:47:52,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:47:52,972 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:47:55,178 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:00,879 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:01,381 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:03,376 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:48:06,377 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:07,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:48:07,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:48:07,979 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:48:09,427 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:12,147 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:17,149 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:17,445 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: partial_history
2023-11-18 12:48:17,447 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:48:17,448 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:48:17,448 DEBUG   SenderThread:45455 [sender.py:send():380] send: metric
2023-11-18 12:48:17,448 DEBUG   SenderThread:45455 [sender.py:send():380] send: history
2023-11-18 12:48:17,449 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: summary_record
2023-11-18 12:48:17,459 INFO    SenderThread:45455 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 12:48:17,494 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/wandb-summary.json
2023-11-18 12:48:19,508 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:22,474 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:22,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:48:22,971 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:48:22,979 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:48:23,530 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/config.yaml
2023-11-18 12:48:28,127 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:31,605 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:33,376 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:48:33,377 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:37,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:48:37,972 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:48:37,980 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:48:39,159 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:39,663 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:44,160 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:49,733 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:48:50,056 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:52,971 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:48:52,972 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:48:52,980 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:48:55,233 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:48:57,792 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:00,242 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:03,378 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:49:05,380 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:07,856 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:08,022 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:49:08,022 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:49:08,023 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:49:11,321 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:15,912 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:16,672 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:21,673 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:23,021 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:49:23,022 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:49:23,022 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:49:27,289 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:30,007 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:33,036 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:33,389 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:49:38,021 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:49:38,022 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:49:38,022 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:49:38,274 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:40,064 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:43,276 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:48,114 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:48,816 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:53,818 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:49:53,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:49:53,939 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:49:53,940 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:49:58,262 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:49:58,988 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:03,380 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:50:03,988 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:06,318 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:08,936 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:50:08,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:50:08,937 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:50:09,231 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:14,234 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:16,394 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:19,934 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:23,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:50:23,938 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:50:23,938 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:50:25,101 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:25,459 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:30,102 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:33,381 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:50:35,570 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:35,722 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:38,936 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:50:38,937 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:50:38,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:50:41,189 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:45,644 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:46,352 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:51,355 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:50:53,718 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:50:53,936 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:50:53,937 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:50:53,937 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:50:57,135 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:03,005 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:03,382 DEBUG   SenderThread:45455 [sender.py:send():380] send: stats
2023-11-18 12:51:03,778 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:51:08,383 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:10,578 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:51:10,579 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:51:10,579 DEBUG   SenderThread:45455 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:51:13,830 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:13,852 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:51:18,831 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:51:23,928 INFO    Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log
2023-11-18 12:51:24,005 DEBUG   HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report