| 2023-11-18 12:43:02,581 INFO StreamThr :45455 [internal.py:wandb_internal():86] W&B internal server running at pid: 45455, started at: 2023-11-18 12:43:02.579911 | |
| 2023-11-18 12:43:02,582 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status | |
| 2023-11-18 12:43:02,587 INFO WriterThread:45455 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_124302-mpnwkerp/run-mpnwkerp.wandb | |
| 2023-11-18 12:43:02,589 DEBUG SenderThread:45455 [sender.py:send():380] send: header | |
| 2023-11-18 12:43:02,589 DEBUG SenderThread:45455 [sender.py:send():380] send: run | |
| 2023-11-18 12:43:03,267 INFO SenderThread:45455 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files | |
| 2023-11-18 12:43:03,276 INFO SenderThread:45455 [sender.py:_start_run_threads():1124] run started: mpnwkerp with start time 1700304182.581382 | |
| 2023-11-18 12:43:03,277 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: check_version | |
| 2023-11-18 12:43:03,277 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: check_version | |
| 2023-11-18 12:43:03,329 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: run_start | |
| 2023-11-18 12:43:03,359 DEBUG HandlerThread:45455 [system_info.py:__init__():32] System info init | |
| 2023-11-18 12:43:03,359 DEBUG HandlerThread:45455 [system_info.py:__init__():47] System info init done | |
| 2023-11-18 12:43:03,360 INFO HandlerThread:45455 [system_monitor.py:start():194] Starting system monitor | |
| 2023-11-18 12:43:03,360 INFO SystemMonitor:45455 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2023-11-18 12:43:03,361 INFO HandlerThread:45455 [system_monitor.py:probe():214] Collecting system info | |
| 2023-11-18 12:43:03,362 INFO SystemMonitor:45455 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-11-18 12:43:03,362 INFO SystemMonitor:45455 [interfaces.py:start():190] Started disk monitoring | |
| 2023-11-18 12:43:03,364 INFO SystemMonitor:45455 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-11-18 12:43:03,365 INFO SystemMonitor:45455 [interfaces.py:start():190] Started memory monitoring | |
| 2023-11-18 12:43:03,366 INFO SystemMonitor:45455 [interfaces.py:start():190] Started network monitoring | |
| 2023-11-18 12:43:03,374 DEBUG HandlerThread:45455 [system_info.py:probe():196] Probing system | |
| 2023-11-18 12:43:03,490 DEBUG HandlerThread:45455 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2023-11-18 12:43:03,490 DEBUG HandlerThread:45455 [system_info.py:probe():244] Probing system done | |
| 2023-11-18 12:43:03,491 DEBUG HandlerThread:45455 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T10:43:03.374908', 'startedAt': '2023-11-18T10:43:02.543043', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=60000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=2', '--logging_steps=20', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.412132263183594}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} | |
| 2023-11-18 12:43:03,491 INFO HandlerThread:45455 [system_monitor.py:probe():224] Finished collecting system info | |
| 2023-11-18 12:43:03,491 INFO HandlerThread:45455 [system_monitor.py:probe():227] Publishing system info | |
| 2023-11-18 12:43:03,491 DEBUG HandlerThread:45455 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |
| 2023-11-18 12:43:03,493 DEBUG HandlerThread:45455 [system_info.py:_save_pip():68] Saving pip packages done | |
| 2023-11-18 12:43:03,495 DEBUG HandlerThread:45455 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |
| 2023-11-18 12:43:04,270 INFO Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/conda-environment.yaml | |
| 2023-11-18 12:43:04,270 INFO Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/requirements.txt | |
| 2023-11-18 12:43:06,591 DEBUG HandlerThread:45455 [system_info.py:_save_conda():87] Saving conda packages done | |
| 2023-11-18 12:43:06,593 INFO HandlerThread:45455 [system_monitor.py:probe():229] Finished publishing system info | |
| 2023-11-18 12:43:06,606 DEBUG SenderThread:45455 [sender.py:send():380] send: files | |
| 2023-11-18 12:43:06,607 INFO SenderThread:45455 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now | |
| 2023-11-18 12:43:06,613 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:43:06,614 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:43:06,614 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:43:06,852 DEBUG SenderThread:45455 [sender.py:send():380] send: telemetry | |
| 2023-11-18 12:43:06,852 DEBUG SenderThread:45455 [sender.py:send():380] send: config | |
| 2023-11-18 12:43:06,853 DEBUG SenderThread:45455 [sender.py:send():380] send: metric | |
| 2023-11-18 12:43:06,853 DEBUG SenderThread:45455 [sender.py:send():380] send: telemetry | |
| 2023-11-18 12:43:06,853 DEBUG SenderThread:45455 [sender.py:send():380] send: metric | |
| 2023-11-18 12:43:06,854 WARNING SenderThread:45455 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) | |
| 2023-11-18 12:43:06,854 DEBUG SenderThread:45455 [sender.py:send():380] send: telemetry | |
| 2023-11-18 12:43:07,206 INFO wandb-upload_0:45455 [upload_job.py:push():131] Uploaded file /tmp/tmplz3npzmzwandb/6cmzyt37-wandb-metadata.json | |
| 2023-11-18 12:43:07,301 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/conda-environment.yaml | |
| 2023-11-18 12:43:07,302 INFO Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/wandb-metadata.json | |
| 2023-11-18 12:43:07,303 INFO Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:43:07,963 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:43:09,312 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:43:11,321 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:43:13,764 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:43:18,765 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:43:21,613 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:43:21,614 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:43:21,614 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:43:23,781 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:43:28,782 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:43:33,787 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:43:34,428 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/config.yaml | |
| 2023-11-18 12:43:36,613 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:43:36,614 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:43:36,655 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:43:39,866 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:43:44,867 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:43:49,868 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:43:51,613 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:43:51,614 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:43:51,655 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:43:55,826 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:00,827 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:03,367 DEBUG SystemMonitor:45455 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2023-11-18 12:44:03,370 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:44:06,371 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:06,614 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:44:06,615 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:44:06,655 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:44:11,830 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:16,831 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:21,615 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:44:21,615 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:44:21,615 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:44:21,875 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:26,876 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:31,876 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:33,371 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:44:36,614 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:44:36,615 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:44:36,665 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:44:37,778 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:42,779 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:47,780 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:51,614 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:44:51,615 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:44:51,665 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:44:52,813 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:44:57,814 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:02,814 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:03,371 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:45:05,778 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:45:06,615 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:45:06,615 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:45:06,616 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:45:07,790 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:45:07,837 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:12,838 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:18,249 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:19,127 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:45:21,614 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:45:21,615 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:45:21,616 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:45:23,798 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:29,756 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:31,459 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:45:33,381 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:45:35,382 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:36,666 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:45:36,668 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:45:36,669 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:45:39,503 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:45:40,804 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:45,806 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:49,578 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:45:51,663 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:51,665 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:45:51,666 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:45:51,666 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:45:56,871 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:45:59,640 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:46:02,660 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:03,372 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:46:06,665 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:46:06,666 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:46:06,666 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:46:07,696 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:46:07,895 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:12,896 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:17,763 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:46:18,617 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:21,666 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:46:21,666 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:46:21,666 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:46:23,904 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:27,817 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:46:29,138 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:33,373 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:46:34,994 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:35,873 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:46:36,665 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:46:36,666 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:46:36,666 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:46:40,890 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:46,708 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:47,941 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:46:51,715 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:46:52,973 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:46:52,974 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:46:52,975 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:46:56,002 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:46:57,218 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:02,219 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:03,374 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:47:06,051 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:47:07,402 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:07,970 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:47:07,971 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:47:07,971 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:47:12,424 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:14,099 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:47:17,425 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:22,970 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:47:22,971 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:47:22,971 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:47:23,216 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:24,154 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:47:28,217 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:33,213 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:47:33,375 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:47:33,375 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:37,971 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:47:37,973 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:47:37,974 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:47:39,132 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:41,268 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:47:44,743 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:49,745 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:47:51,327 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:47:52,971 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:47:52,971 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:47:52,972 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:47:55,178 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:00,879 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:01,381 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:48:03,376 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:48:06,377 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:07,971 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:48:07,971 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:48:07,979 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:48:09,427 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:48:12,147 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:17,149 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:17,445 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: partial_history | |
| 2023-11-18 12:48:17,447 DEBUG SenderThread:45455 [sender.py:send():380] send: metric | |
| 2023-11-18 12:48:17,448 DEBUG SenderThread:45455 [sender.py:send():380] send: metric | |
| 2023-11-18 12:48:17,448 DEBUG SenderThread:45455 [sender.py:send():380] send: metric | |
| 2023-11-18 12:48:17,448 DEBUG SenderThread:45455 [sender.py:send():380] send: history | |
| 2023-11-18 12:48:17,449 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: summary_record | |
| 2023-11-18 12:48:17,459 INFO SenderThread:45455 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end | |
| 2023-11-18 12:48:17,494 INFO Thread-12 :45455 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/wandb-summary.json | |
| 2023-11-18 12:48:19,508 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:48:22,474 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:22,971 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:48:22,971 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:48:22,979 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:48:23,530 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/config.yaml | |
| 2023-11-18 12:48:28,127 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:31,605 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:48:33,376 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:48:33,377 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:37,971 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:48:37,972 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:48:37,980 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:48:39,159 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:39,663 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:48:44,160 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:49,733 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:48:50,056 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:52,971 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:48:52,972 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:48:52,980 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:48:55,233 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:48:57,792 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:49:00,242 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:03,378 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:49:05,380 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:07,856 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:49:08,022 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:49:08,022 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:49:08,023 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:49:11,321 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:15,912 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:49:16,672 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:21,673 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:23,021 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:49:23,022 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:49:23,022 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:49:27,289 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:30,007 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:49:33,036 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:33,389 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:49:38,021 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:49:38,022 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:49:38,022 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:49:38,274 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:40,064 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:49:43,276 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:48,114 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:49:48,816 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:53,818 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:49:53,937 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:49:53,939 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:49:53,940 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:49:58,262 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:49:58,988 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:03,380 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:50:03,988 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:06,318 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:50:08,936 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:50:08,937 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:50:08,937 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:50:09,231 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:14,234 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:16,394 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:50:19,934 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:23,937 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:50:23,938 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:50:23,938 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:50:25,101 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:25,459 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:50:30,102 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:33,381 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:50:35,570 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:50:35,722 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:38,936 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:50:38,937 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:50:38,937 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:50:41,189 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:45,644 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:50:46,352 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:51,355 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:50:53,718 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:50:53,936 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:50:53,937 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:50:53,937 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:50:57,135 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:51:03,005 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:51:03,382 DEBUG SenderThread:45455 [sender.py:send():380] send: stats | |
| 2023-11-18 12:51:03,778 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:51:08,383 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:51:10,578 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:51:10,579 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:51:10,579 DEBUG SenderThread:45455 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:51:13,830 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:51:13,852 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:51:18,831 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:51:23,928 INFO Thread-12 :45455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_124302-mpnwkerp/files/output.log | |
| 2023-11-18 12:51:24,005 DEBUG HandlerThread:45455 [handler.py:handle_request():146] handle_request: status_report | |