2023-11-18 13:20:20,916 INFO StreamThr :60204 [internal.py:wandb_internal():86] W&B internal server running at pid: 60204, started at: 2023-11-18 13:20:20.914208 2023-11-18 13:20:20,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status 2023-11-18 13:20:20,922 INFO WriterThread:60204 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_132020-qg88991p/run-qg88991p.wandb 2023-11-18 13:20:20,923 DEBUG SenderThread:60204 [sender.py:send():380] send: header 2023-11-18 13:20:20,924 DEBUG SenderThread:60204 [sender.py:send():380] send: run 2023-11-18 13:20:21,360 INFO SenderThread:60204 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_132020-qg88991p/files 2023-11-18 13:20:21,369 INFO SenderThread:60204 [sender.py:_start_run_threads():1124] run started: qg88991p with start time 1700306420.91509 2023-11-18 13:20:21,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: check_version 2023-11-18 13:20:21,370 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: check_version 2023-11-18 13:20:21,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: run_start 2023-11-18 13:20:21,425 DEBUG HandlerThread:60204 [system_info.py:__init__():32] System info init 2023-11-18 13:20:21,425 DEBUG HandlerThread:60204 [system_info.py:__init__():47] System info init done 2023-11-18 13:20:21,425 INFO HandlerThread:60204 [system_monitor.py:start():194] Starting system monitor 2023-11-18 13:20:21,425 INFO SystemMonitor:60204 [system_monitor.py:_start():158] Starting system asset monitoring threads 2023-11-18 13:20:21,426 INFO HandlerThread:60204 [system_monitor.py:probe():214] Collecting system info 2023-11-18 13:20:21,426 INFO SystemMonitor:60204 [interfaces.py:start():190] Started cpu monitoring 2023-11-18 13:20:21,428 INFO SystemMonitor:60204 [interfaces.py:start():190] Started disk monitoring 2023-11-18 13:20:21,429 INFO SystemMonitor:60204 [interfaces.py:start():190] Started gpu monitoring 2023-11-18 13:20:21,430 INFO SystemMonitor:60204 [interfaces.py:start():190] Started memory monitoring 2023-11-18 13:20:21,431 INFO SystemMonitor:60204 [interfaces.py:start():190] Started network monitoring 2023-11-18 13:20:21,436 DEBUG HandlerThread:60204 [system_info.py:probe():196] Probing system 2023-11-18 13:20:21,544 DEBUG HandlerThread:60204 [gitlib.py:_init_repo():56] git repository is invalid 2023-11-18 13:20:21,545 DEBUG HandlerThread:60204 [system_info.py:probe():244] Probing system done 2023-11-18 13:20:21,545 DEBUG HandlerThread:60204 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T11:20:21.437109', 'startedAt': '2023-11-18T11:20:20.879359', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=60000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=2', '--logging_steps=20', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--torch_compile', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub', '--push_to_hub_model_id=WhisperLargeFinnishV3'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.43611145019531}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} 2023-11-18 13:20:21,545 INFO HandlerThread:60204 [system_monitor.py:probe():224] Finished collecting system info 2023-11-18 13:20:21,545 INFO HandlerThread:60204 [system_monitor.py:probe():227] Publishing system info 2023-11-18 13:20:21,545 DEBUG HandlerThread:60204 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2023-11-18 13:20:21,548 DEBUG HandlerThread:60204 [system_info.py:_save_pip():68] Saving pip packages done 2023-11-18 13:20:21,548 DEBUG HandlerThread:60204 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment 2023-11-18 13:20:22,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/requirements.txt 2023-11-18 13:20:22,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/conda-environment.yaml 2023-11-18 13:20:23,606 DEBUG HandlerThread:60204 [system_info.py:_save_conda():87] Saving conda packages done 2023-11-18 13:20:23,609 INFO HandlerThread:60204 [system_monitor.py:probe():229] Finished publishing system info 2023-11-18 13:20:23,613 DEBUG SenderThread:60204 [sender.py:send():380] send: files 2023-11-18 13:20:23,613 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now 2023-11-18 13:20:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:20:23,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:20:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:20:23,834 DEBUG SenderThread:60204 [sender.py:send():380] send: telemetry 2023-11-18 13:20:23,835 DEBUG SenderThread:60204 [sender.py:send():380] send: config 2023-11-18 13:20:23,835 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:20:23,836 DEBUG SenderThread:60204 [sender.py:send():380] send: telemetry 2023-11-18 13:20:23,836 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:20:23,836 WARNING SenderThread:60204 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) 2023-11-18 13:20:23,837 DEBUG SenderThread:60204 [sender.py:send():380] send: telemetry 2023-11-18 13:20:24,224 INFO wandb-upload_0:60204 [upload_job.py:push():131] Uploaded file /tmp/tmphe77gjs0wandb/1613lfeo-wandb-metadata.json 2023-11-18 13:20:24,373 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/conda-environment.yaml 2023-11-18 13:20:24,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-metadata.json 2023-11-18 13:20:24,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:20:26,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:26,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:20:28,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:20:31,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:36,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:38,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:20:38,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:20:38,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:20:41,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:46,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:51,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:52,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/config.yaml 2023-11-18 13:20:53,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:20:53,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:20:53,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:20:57,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:02,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:07,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:08,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:21:08,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:21:08,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:21:12,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:17,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:21,431 DEBUG SystemMonitor:60204 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2023-11-18 13:21:21,433 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:21:23,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:21:23,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:21:23,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:21:28,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:33,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:38,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:21:38,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:21:38,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:21:38,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:43,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:48,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:51,433 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:21:53,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:21:53,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:21:53,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:21:53,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:58,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:03,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:08,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:22:08,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:22:08,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:22:09,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:14,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:19,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:21,434 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:22:22,865 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:22:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:22:23,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:22:23,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:22:24,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:24,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:22:29,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:34,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:39,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:22:39,631 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:22:39,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:22:39,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:44,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:22:44,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:49,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:51,435 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:22:52,735 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:22:53,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:22:53,623 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:22:55,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:00,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:04,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:06,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:09,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:23:09,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:23:09,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:23:11,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:12,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:16,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:21,435 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:23:22,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:22,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:24,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:23:24,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:23:24,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:23:27,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:33,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:33,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:38,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:39,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:23:39,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:23:39,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:23:42,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:43,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:48,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:50,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:51,436 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:23:54,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:23:54,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:23:54,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:23:54,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:00,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:00,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:05,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:09,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:24:09,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:24:09,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:24:10,284 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:10,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:15,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:18,335 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:20,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:21,436 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:24:24,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:24:24,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:24:24,743 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:24:25,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:28,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:30,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:36,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:36,443 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:39,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:24:39,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:24:39,743 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:24:41,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:46,495 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:47,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:51,438 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:24:53,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:54,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:24:54,743 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:24:56,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:58,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:04,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:04,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:09,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:09,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:25:09,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:25:09,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:25:14,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:14,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:19,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:21,438 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:25:22,702 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:24,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:24,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:25:24,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:25:29,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:32,764 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:35,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:39,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:25:39,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:25:39,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:25:40,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:40,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:25:40,128 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:25:40,129 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:25:40,129 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:25:40,129 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:25:40,130 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:25:40,141 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:25:40,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:25:42,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:45,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:50,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:51,439 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:25:52,888 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:54,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:25:54,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:25:55,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:00,945 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:01,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:06,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:06,981 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/config.yaml 2023-11-18 13:26:09,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:26:09,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:26:09,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:26:11,004 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:11,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:16,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:21,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:21,439 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:26:22,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:26,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:26:26,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:26:26,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:26:27,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:29,156 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:32,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:37,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:41,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:26:41,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:26:41,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:26:41,234 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:43,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:49,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:49,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:51,451 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:26:54,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:56,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:26:56,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:26:56,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:26:58,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:00,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:06,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:06,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:11,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:27:11,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:27:11,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:16,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:16,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:21,440 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:27:22,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:26,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:27:26,210 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:27:26,541 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:28,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:33,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:36,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:38,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:42,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:27:42,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:27:42,683 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:27:43,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:44,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:48,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:51,442 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:27:54,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:54,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:57,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:27:57,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:27:57,681 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:27:59,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:04,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:05,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:10,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:12,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:28:12,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:28:12,681 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:28:12,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:15,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:21,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:21,443 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:28:22,892 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:26,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:28,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:28:28,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:28:28,200 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:28:30,949 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:32,348 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:37,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:41,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:42,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:43,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:28:43,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:28:43,198 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:28:47,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:47,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:28:47,706 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:28:47,707 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:28:47,710 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:28:48,067 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:28:49,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:51,442 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:28:53,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:58,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:28:58,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:28:58,199 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:28:59,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:59,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:04,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:07,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:10,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:13,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:29:13,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:29:13,208 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:29:15,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:17,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:20,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:21,444 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:29:26,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:27,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:28,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:29:28,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:29:28,207 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:29:32,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:38,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:39,419 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:43,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:29:43,208 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:29:43,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:49,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:49,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:51,444 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:29:54,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:56,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:58,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:29:58,206 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:29:59,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:04,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:06,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:09,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:13,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:30:13,206 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:30:14,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:15,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:20,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:21,444 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:30:22,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:25,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:28,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:30:28,207 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:30:31,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:34,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:37,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:42,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:43,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:30:43,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:30:43,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:30:46,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:47,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:51,446 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:30:53,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:54,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:58,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:30:58,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:30:58,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:04,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:05,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:09,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:13,059 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:13,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:31:13,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:31:14,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:19,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:21,446 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:31:23,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:25,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:28,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:31:28,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:31:30,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:33,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:35,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:40,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:44,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:31:44,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:31:44,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:31:45,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:47,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:50,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:51,447 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:31:56,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:57,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:59,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:31:59,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:31:59,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:32:02,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:04,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:32:04,199 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:32:04,199 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:32:04,202 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:32:04,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:32:05,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:07,426 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:08,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:14,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:14,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:32:14,430 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:32:15,481 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:19,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:21,448 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:32:25,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:25,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:29,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:32:29,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:32:29,430 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:32:30,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:35,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:35,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:40,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:43,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:44,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:32:44,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:32:44,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:32:46,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:51,448 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:32:52,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:53,735 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:57,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:59,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:32:59,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:32:59,432 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:33:00,791 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:02,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:07,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:10,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:13,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:14,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:33:14,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:33:18,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:20,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:21,449 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:33:24,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:28,979 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:29,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:33:29,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:33:29,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:34,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:39,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:40,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:45,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:45,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:33:45,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:33:45,489 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:33:49,193 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:50,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:51,449 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:33:55,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:57,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:00,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:34:00,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:34:00,488 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:34:01,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:07,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:07,321 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:12,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:15,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:34:15,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:34:15,487 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:34:17,399 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:17,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:21,452 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:34:23,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:27,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:28,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:30,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:34:30,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:34:30,488 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:34:33,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:35,510 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:38,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:43,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:45,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:34:45,488 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:34:45,567 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:49,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:51,453 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:34:55,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:55,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:00,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:01,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:35:01,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:35:01,220 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:35:03,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:06,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:11,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:35:11,141 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:35:11,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:35:11,144 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:35:11,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:35:12,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:13,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:16,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:35:16,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:35:17,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:21,453 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:35:21,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:22,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:27,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:31,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:35:31,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:35:31,890 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:33,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:38,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:41,960 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:44,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:47,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:35:47,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:35:47,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:35:49,007 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:49,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:51,453 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:35:54,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:59,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:00,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:02,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:36:02,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:36:06,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:07,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:11,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:16,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:17,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:36:17,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:36:19,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:21,454 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:36:21,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:26,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:27,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:31,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:32,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:36:32,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:36:37,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:42,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:43,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:47,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:36:47,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:36:48,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:51,455 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:36:53,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:53,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:58,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:01,477 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:02,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:37:02,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:37:04,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:10,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:11,545 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:15,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:17,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:37:17,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:37:17,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:37:21,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:21,455 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:37:21,602 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:26,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:31,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:32,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:32,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:37:32,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:37:32,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:37:37,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:39,743 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:42,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:47,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:37:47,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:37:48,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:49,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:51,456 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:37:53,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:57,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:59,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:02,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:38:02,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:38:02,170 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:38:04,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:07,933 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:09,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:14,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:17,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:38:17,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:38:18,003 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:20,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:21,458 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:38:24,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:38:24,186 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:38:24,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:38:24,188 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:38:25,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:38:25,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:26,057 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:26,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:31,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:32,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:38:32,170 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:38:36,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:36,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:41,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:44,167 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:46,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:47,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:38:47,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:38:51,458 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:38:52,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:54,231 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:57,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:02,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:39:02,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:39:03,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:03,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:08,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:13,370 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:14,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:17,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:39:17,171 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:39:19,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:21,422 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:21,458 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:39:24,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:30,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:31,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:32,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:39:32,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:39:35,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:40,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:41,556 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:45,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:47,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:39:47,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:39:47,911 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:39:51,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:51,459 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:39:51,613 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:56,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:59,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:02,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:02,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:40:02,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:40:08,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:09,721 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:13,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:17,776 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:17,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:40:17,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:40:18,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:21,459 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:40:23,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:25,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:28,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:33,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:34,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:40:34,188 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:40:37,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:39,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:45,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:45,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:49,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:40:49,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:40:50,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:51,461 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:40:55,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:56,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:00,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:04,130 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:04,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:41:04,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:41:06,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:11,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:14,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:16,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:19,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:41:19,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:41:21,462 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:41:22,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:24,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:27,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:31,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:41:31,482 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:41:31,482 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:41:31,484 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:41:32,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:41:33,310 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:33,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:34,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:41:34,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:41:34,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:39,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:42,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:45,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:49,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:41:49,188 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:41:50,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:51,462 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:41:51,713 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:55,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:59,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:00,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:04,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:42:04,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:42:06,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:11,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:15,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:17,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:19,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:42:19,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:42:21,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:42:22,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:23,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:28,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:33,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:34,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:34,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:42:34,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:42:38,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:43,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:44,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:48,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:49,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:42:49,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:42:49,998 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:42:51,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:42:52,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:54,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:59,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:04,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:04,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:04,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:43:04,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:43:04,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:43:10,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:12,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:15,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:19,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:43:19,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:43:19,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:43:21,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:21,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:43:22,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:26,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:30,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:31,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:36,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:43:36,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:43:36,272 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:43:37,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:40,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:42,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:47,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:51,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:43:51,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:43:51,273 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:43:51,594 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:43:52,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:54,587 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:57,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:01,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:03,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:06,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:44:06,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:44:06,274 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:44:08,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:09,691 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:14,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:19,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:19,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:21,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:44:21,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:44:21,272 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:44:21,596 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:44:25,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:27,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:30,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:35,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:37,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:44:37,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:44:37,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:44:39,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:40,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:45,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:47,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:44:47,713 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:44:47,713 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:44:47,716 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:44:47,934 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:44:49,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:51,466 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:44:51,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:52,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:44:52,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:44:52,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:44:56,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:58,134 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:01,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:06,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:07,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:45:07,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:45:07,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:45:08,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:11,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:17,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:18,255 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:21,466 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:45:22,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:22,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:45:22,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:45:22,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:45:27,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:28,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:32,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:37,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:45:37,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:45:37,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:38,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:42,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:47,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:50,534 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:51,467 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:45:52,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:45:52,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:45:53,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:59,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:00,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:04,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:07,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:46:07,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:46:07,719 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:46:09,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:10,665 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:14,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:18,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:20,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:21,467 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:46:22,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:46:22,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:46:25,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:29,827 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:31,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:36,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:37,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:46:37,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:46:37,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:41,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:46,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:47,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:51,468 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:46:52,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:53,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:46:53,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:46:53,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:46:55,992 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:57,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:02,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:06,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:07,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:08,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:47:08,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:47:13,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:16,114 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:19,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:21,480 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:47:23,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:47:23,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:47:23,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:47:24,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:24,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:29,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:32,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:35,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:39,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:47:39,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:47:39,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:47:41,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:42,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:46,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:50,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:51,469 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:47:51,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:54,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:47:54,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:47:54,134 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:47:57,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:00,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:48:00,576 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:48:00,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:48:00,579 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:48:01,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:48:02,407 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:02,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:07,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:09,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:48:09,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:48:09,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:48:10,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:13,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:18,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:20,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:21,470 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:48:24,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:48:24,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:24,133 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:48:28,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:29,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:34,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:38,745 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:39,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:48:39,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:48:40,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:45,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:46,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:50,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:51,471 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:48:55,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:48:55,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:48:55,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:48:55,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:58,890 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:01,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:06,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:07,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:10,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:49:10,393 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:49:12,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:16,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:18,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:21,472 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:49:23,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:24,231 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:25,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:49:25,384 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:49:28,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:34,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:34,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:39,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:40,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:49:40,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:49:40,390 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:49:44,373 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:45,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:50,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:51,474 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:49:54,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:55,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:55,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:49:55,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:00,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:02,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:05,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:10,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:50:10,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:11,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:12,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:16,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:20,676 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:21,473 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:50:22,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:25,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:50:25,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:27,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:30,726 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:33,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:38,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:40,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:50:40,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:40,789 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:43,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:48,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:49,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:51,474 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:50:55,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:50:55,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:50:55,402 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:55,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:58,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:01,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:05,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:51:05,402 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:51:05,403 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:51:05,406 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:51:05,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:51:06,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:07,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:07,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:10,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:51:10,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:51:12,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:17,041 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:18,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:21,475 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:51:24,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:25,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:51:25,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:51:25,402 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:51:27,095 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:29,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:34,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:35,147 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:39,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:41,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:51:41,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:51:41,235 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:51:45,208 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:45,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:50,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:51,476 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:51:52,250 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:55,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:56,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:51:56,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:01,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:02,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:06,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:11,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:52:11,234 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:12,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:16,401 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:17,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:21,477 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:52:23,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:24,451 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:26,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:52:26,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:29,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:34,514 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:35,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:40,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:41,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:52:41,234 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:42,573 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:46,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:51,477 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:52:51,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:52,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:56,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:52:56,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:52:56,236 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:56,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:00,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:01,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:06,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:10,748 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:11,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:53:11,234 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:53:12,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:17,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:20,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:21,478 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:53:22,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:26,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:53:26,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:53:26,239 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:53:28,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:28,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:33,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:38,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:38,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:41,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:53:41,235 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:53:43,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:46,984 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:49,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:51,478 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:53:54,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:56,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:53:56,235 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:53:57,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:59,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:04,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:07,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:09,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:12,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:54:12,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:54:13,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:54:13,506 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:54:13,507 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:54:13,509 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:54:14,242 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:54:14,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:15,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:15,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:20,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:21,480 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:54:25,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:26,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:27,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:54:27,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:54:31,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:33,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:36,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:42,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:42,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:54:42,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:54:42,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:47,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:51,480 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:54:52,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:52,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:57,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:54:57,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:54:57,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:02,580 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:02,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:07,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:10,642 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:12,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:55:12,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:55:13,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:18,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:20,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:21,480 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:55:24,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:27,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:55:27,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:55:28,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:29,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:34,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:38,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:39,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:42,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:55:42,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:55:45,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:48,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:51,481 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:55:51,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:56,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:56,993 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:57,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:55:57,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:56:02,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:07,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:07,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:12,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:56:12,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:56:12,460 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:56:13,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:15,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:18,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:21,482 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:56:24,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:25,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:27,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:56:27,458 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:56:29,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:35,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:35,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:40,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:42,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:56:42,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:56:45,394 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:45,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:50,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:51,483 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:56:55,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:56,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:57,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:56:57,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:57:02,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:03,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:07,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:12,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:12,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:57:12,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:57:13,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:17,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:20,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:57:20,739 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:57:20,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:57:20,742 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:57:21,483 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:57:21,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:57:22,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:23,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:28,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:28,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:57:28,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:57:28,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:57:32,705 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:34,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:40,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:40,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:43,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:57:43,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:57:43,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:57:46,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:48,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:51,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:51,484 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:57:56,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:58,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:58,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:57:58,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:57:58,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:58:02,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:07,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:08,944 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:12,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:15,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:58:15,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:58:15,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:58:18,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:19,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:21,484 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:58:23,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:27,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:28,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:30,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:58:30,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:58:33,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:37,123 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:39,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:45,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:45,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:58:45,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:45,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:58:50,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:51,485 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:58:55,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:56,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:00,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:59:00,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:59:00,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:59:01,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:03,301 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:06,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:11,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:13,367 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:15,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:59:15,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:59:16,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:21,485 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:59:22,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:23,433 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:27,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:30,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:59:30,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:59:32,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:35,514 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:38,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:43,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:45,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:59:45,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:59:47,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:48,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:51,486 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:59:53,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:55,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:58,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:01,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:00:01,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:00:01,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:00:04,503 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:05,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:09,504 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:14,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:15,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:16,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:00:16,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:00:20,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:21,487 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:00:23,837 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:25,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:30,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:31,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:00:31,106 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:00:31,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:00:31,109 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:00:31,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:00:31,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:00:31,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:00:32,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:36,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:41,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:42,967 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:46,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:00:46,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:00:47,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:51,022 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:51,487 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:00:53,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:58,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:01,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:01,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:01:01,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:01:03,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:08,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:11,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:14,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:17,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:01:17,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:01:17,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:01:19,210 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:19,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:21,488 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:01:24,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:29,277 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:30,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:32,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:01:32,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:01:32,185 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:01:35,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:37,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:41,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:47,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:47,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:01:47,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:01:47,185 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:01:47,410 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:51,489 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:01:52,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:55,472 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:58,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:03,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:03,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:02:03,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:02:03,357 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:02:05,540 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:08,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:14,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:15,608 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:18,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:02:18,355 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:02:19,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:21,489 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:02:23,662 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:25,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:30,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:33,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:02:33,358 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:02:33,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:35,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:40,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:41,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:45,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:49,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:02:49,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:02:49,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:02:51,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:51,490 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:02:51,841 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:56,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:01,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:01,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:04,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:03:04,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:03:04,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:03:07,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:09,963 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:12,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:18,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:19,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:03:19,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:03:19,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:03:20,019 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:21,490 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:03:23,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:28,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:29,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:34,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:35,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:03:35,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:03:35,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:03:36,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:03:36,575 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:03:36,575 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:03:36,578 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:03:37,134 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:03:37,135 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:39,142 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:39,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:44,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:47,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:49,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:50,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:03:50,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:03:51,492 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:03:55,249 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:55,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:00,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:05,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:04:05,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:04:05,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:06,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:11,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:15,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:16,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:20,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:04:20,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:04:20,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:04:21,493 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:04:22,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:25,427 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:28,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:33,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:35,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:04:35,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:04:35,494 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:38,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:44,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:45,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:49,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:50,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:04:50,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:04:50,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:04:51,493 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:04:53,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:55,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:00,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:03,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:05,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:05:05,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:05,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:05:11,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:11,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:16,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:20,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:05:20,990 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:05:21,493 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:05:21,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:21,802 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:26,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:29,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:32,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:37,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:37,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:05:37,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:05:37,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:05:39,923 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:42,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:48,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:49,989 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:51,494 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:05:52,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:05:52,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:05:52,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:05:53,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:58,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:59,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:04,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:07,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:06:07,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:06:07,421 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:06:09,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:10,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:14,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:18,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:20,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:21,495 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:06:22,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:06:22,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:06:22,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:06:26,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:28,233 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:31,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:36,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:37,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:06:37,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:06:38,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:42,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:44,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:06:44,758 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:06:44,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:06:44,761 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:06:45,349 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:06:45,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:47,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:47,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:51,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:06:52,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:06:52,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:06:52,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:06:53,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:55,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:58,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:03,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:03,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:07,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:07:07,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:07:07,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:07:09,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:13,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:15,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:20,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:21,499 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:07:22,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:07:22,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:07:23,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:25,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:30,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:33,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:36,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:37,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:07:37,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:07:37,436 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:07:41,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:43,749 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:47,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:51,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:07:52,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:07:52,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:07:52,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:53,812 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:57,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:02,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:03,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:07,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:08:07,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:08:08,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:11,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:14,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:20,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:21,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:08:22,030 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:22,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:08:22,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:08:22,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:08:25,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:30,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:32,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:35,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:37,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:08:37,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:08:37,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:08:41,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:44,264 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:47,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:51,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:08:52,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:52,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:08:52,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:08:52,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:57,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:02,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:04,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:07,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:09:07,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:09:07,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:12,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:14,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:17,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:21,499 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:09:22,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:09:22,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:09:23,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:24,522 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:28,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:33,583 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:34,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:37,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:09:37,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:09:39,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:43,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:45,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:50,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:51,500 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:09:52,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:09:52,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:09:53,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:55,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:00,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:01,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:10:01,036 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:10:01,037 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:10:01,039 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:10:01,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:10:01,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:03,778 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:06,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:08,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:10:08,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:10:08,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:10:11,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:11,827 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:16,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:21,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:21,501 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:10:23,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:10:23,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:10:23,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:10:23,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:26,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:31,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:33,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:36,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:38,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:10:38,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:10:38,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:10:42,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:47,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:51,502 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:10:52,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:53,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:10:53,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:10:53,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:10:54,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:57,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:02,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:04,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:08,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:08,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:11:08,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:11:08,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:11:13,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:14,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:18,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:21,503 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:11:23,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:23,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:11:23,801 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:11:23,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:11:26,255 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:28,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:34,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:34,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:38,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:11:38,800 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:11:40,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:45,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:46,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:50,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:51,503 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:11:55,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:11:55,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:11:55,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:11:56,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:58,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:01,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:06,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:06,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:10,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:12:10,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:12:12,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:14,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:18,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:21,504 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:12:24,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:24,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:25,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:12:25,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:12:25,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:12:29,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:34,715 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:35,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:40,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:12:40,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:12:40,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:12:41,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:45,795 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:46,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:51,504 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:12:52,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:55,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:12:55,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:12:55,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:12:57,863 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:57,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:02,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:05,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:08,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:10,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:13:10,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:13:10,122 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:13:13,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:17,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:18,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:21,505 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:13:23,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:26,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:13:26,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:13:26,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:13:28,027 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:28,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:33,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:38,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:38,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:13:38,894 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:13:38,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:13:38,899 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:13:39,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:13:40,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:41,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:13:41,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:13:43,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:48,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:50,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:51,505 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:13:54,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:56,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:13:56,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:13:56,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:13:58,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:59,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:04,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:09,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:11,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:14:11,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:14:11,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:14:12,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:15,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:20,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:21,506 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:14:22,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:26,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:14:26,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:14:26,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:26,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:14:31,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:34,460 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:37,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:41,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:14:41,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:14:41,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:14:42,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:46,534 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:48,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:51,507 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:14:54,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:54,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:56,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:14:56,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:14:59,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:04,660 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:04,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:09,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:11,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:15:11,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:15:11,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:15:12,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:15,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:20,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:21,507 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:15:22,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:26,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:26,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:15:26,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:15:31,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:34,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:37,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:41,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:15:41,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:15:41,327 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:15:42,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:46,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:47,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:51,507 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:15:53,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:56,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:15:56,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:15:57,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:59,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:05,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:06,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:10,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:12,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:16:12,157 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:16:14,116 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:16,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:21,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:21,508 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:16:26,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:27,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:16:27,155 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:16:28,208 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:31,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:36,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:38,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:41,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:42,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:16:42,156 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:16:47,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:50,326 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:51,508 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:16:52,509 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:57,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:16:57,156 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:16:58,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:58,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:03,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:05,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:17:05,949 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:17:05,949 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:17:05,951 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:17:06,441 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:17:08,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:08,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:13,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:17:13,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:17:13,444 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:17:14,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:16,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:19,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:21,509 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:17:24,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:24,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:28,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:17:28,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:17:28,463 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:17:30,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:35,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:36,678 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:40,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:43,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:17:43,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:17:43,464 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:17:46,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:48,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:51,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:51,510 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:17:56,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:56,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:58,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:17:58,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:17:58,463 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:18:02,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:04,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:08,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:13,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:18:13,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:18:13,463 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:18:13,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:14,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:18,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:21,511 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:18:24,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:28,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:18:28,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:18:28,464 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:18:29,062 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:29,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:34,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:40,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:41,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:43,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:18:43,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:18:43,464 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:18:45,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:51,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:51,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:51,511 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:18:56,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:58,254 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:58,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:18:58,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:18:58,464 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:19:01,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:07,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:08,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:12,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:14,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:19:14,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:19:14,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:19:16,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:18,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:21,512 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:19:23,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:28,455 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:29,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:19:29,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:19:29,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:19:29,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:34,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:38,515 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:40,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:44,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:19:44,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:19:44,196 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:19:45,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:48,582 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:51,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:51,512 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:19:56,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:59,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:19:59,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:19:59,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:20:00,655 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:02,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:07,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:08,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:12,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:15,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:20:15,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:20:15,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:20:18,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:18,772 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:21,512 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:20:24,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:29,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:29,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:20:29,872 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:20:29,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:20:29,875 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:20:30,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:20:30,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:20:30,514 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:20:30,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:20:30,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:31,853 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:34,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:40,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:40,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:45,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:45,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:20:45,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:20:45,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:20:51,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:51,513 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:20:52,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:56,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:00,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:21:00,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:21:00,515 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:21:01,030 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:01,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:06,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:12,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:13,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:15,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:21:15,515 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:21:17,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:21,514 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:21:22,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:23,163 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:27,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:30,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:21:30,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:21:30,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:21:33,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:35,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:38,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:44,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:45,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:45,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:21:45,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:21:45,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:21:49,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:51,526 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:21:55,348 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:57,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:00,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:00,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:22:00,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:22:00,519 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:22:05,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:05,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:10,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:15,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:22:15,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:22:15,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:16,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:20,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:21,516 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:22:24,535 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:25,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:30,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:22:30,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:22:30,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:36,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:36,608 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:41,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:44,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:45,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:22:45,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:22:46,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:51,517 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:22:52,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:52,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:57,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:00,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:23:00,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:23:02,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:06,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:08,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:13,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:15,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:23:15,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:23:16,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:18,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:21,517 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:23:24,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:26,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:30,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:30,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:23:30,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:23:35,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:39,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:41,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:45,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:23:45,519 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:23:46,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:49,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:51,517 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:23:52,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:57,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:58,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:23:58,201 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:23:58,201 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:23:58,203 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:23:59,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:23:59,193 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:00,198 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:00,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:24:00,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:24:02,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:07,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:09,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:13,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:15,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:24:15,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:24:15,919 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:24:19,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:19,359 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:21,518 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:24:24,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:27,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:29,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:30,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:24:30,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:24:30,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:24:35,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:37,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:40,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:45,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:45,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:24:45,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:24:45,918 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:24:47,548 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:51,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:51,518 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:24:56,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:59,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:00,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:25:00,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:25:02,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:07,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:11,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:12,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:15,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:25:15,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:25:18,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:20,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:21,519 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:25:23,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:28,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:30,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:30,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:25:30,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:25:30,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:25:34,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:38,883 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:39,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:44,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:46,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:25:46,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:25:46,655 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:25:48,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:49,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:51,519 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:25:55,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:59,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:01,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:01,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:26:01,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:26:01,655 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:26:06,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:11,095 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:12,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:17,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:18,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:26:18,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:26:18,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:26:21,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:21,520 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:26:23,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:28,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:29,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:33,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:26:33,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:26:33,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:26:34,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:37,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:39,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:44,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:48,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:26:48,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:26:48,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:26:49,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:49,351 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:51,520 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:26:54,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:59,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:01,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:03,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:27:03,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:27:03,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:27:05,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:09,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:10,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:15,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:18,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:27:18,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:27:18,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:27:19,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:27:19,016 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:27:19,016 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:27:19,018 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:27:19,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:27:20,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:21,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:21,520 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:27:21,577 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:26,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:29,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:31,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:33,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:27:33,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:27:33,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:27:37,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:41,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:42,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:47,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:48,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:27:48,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:27:48,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:27:49,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:51,521 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:27:53,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:58,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:59,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:03,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:28:03,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:28:03,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:28:04,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:09,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:10,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:14,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:18,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:28:18,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:28:18,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:28:18,999 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:20,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:21,521 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:28:25,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:27,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:31,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:33,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:28:33,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:28:33,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:28:36,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:37,120 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:41,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:46,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:47,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:48,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:28:48,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:28:48,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:28:51,532 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:28:52,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:57,288 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:58,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:03,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:29:03,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:29:03,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:29:03,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:08,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:13,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:13,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:18,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:29:18,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:29:18,100 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:29:19,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:21,522 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:29:23,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:24,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:29,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:33,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:29:33,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:29:33,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:29:35,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:35,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:40,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:46,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:47,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:48,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:29:48,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:29:48,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:29:51,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:51,523 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:29:55,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:56,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:01,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:03,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:30:03,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:30:03,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:30:03,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:07,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:12,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:13,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:17,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:18,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:30:18,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:30:18,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:30:21,524 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:30:22,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:23,783 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:27,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:32,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:33,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:30:33,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:30:33,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:30:35,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:38,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:43,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:45,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:30:45,787 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:30:45,788 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:30:45,791 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:30:45,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:30:46,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:47,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:48,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:30:48,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:30:48,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:51,525 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:30:53,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:56,016 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:58,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:03,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:31:03,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:31:04,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:06,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:09,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:15,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:16,165 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:18,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:31:18,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:31:21,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:21,526 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:31:25,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:26,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:31,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:33,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:31:33,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:31:33,682 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:31:35,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:36,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:41,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:47,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:47,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:48,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:31:48,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:31:48,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:31:51,526 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:31:52,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:57,432 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:58,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:03,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:03,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:32:03,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:32:03,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:32:05,481 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:08,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:13,531 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:14,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:18,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:32:18,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:32:18,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:32:19,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:21,528 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:32:25,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:25,606 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:30,509 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:34,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:32:34,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:32:34,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:32:35,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:37,678 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:40,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:45,804 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:46,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:49,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:32:49,714 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:32:51,527 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:32:52,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:55,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:57,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:02,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:04,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:33:04,714 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:33:05,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:07,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:13,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:14,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:18,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:19,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:33:19,714 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:33:21,528 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:33:23,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:26,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:29,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:34,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:34,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:33:34,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:33:36,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:40,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:44,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:45,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:49,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:33:49,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:33:50,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:51,531 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:33:54,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:56,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:01,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:01,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:34:01,633 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:34:01,633 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:34:01,636 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:34:02,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:34:03,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:04,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:04,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:34:04,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:34:06,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:11,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:13,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:17,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:19,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:34:19,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:34:21,531 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:34:23,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:23,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:28,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:33,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:34,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:34:34,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:34:35,539 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:38,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:44,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:45,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:49,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:34:49,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:34:49,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:51,531 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:34:55,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:59,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:01,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:04,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:35:04,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:35:06,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:09,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:12,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:17,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:19,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:35:19,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:35:19,781 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:21,532 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:35:23,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:29,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:29,847 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:34,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:35,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:35:35,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:35:35,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:35:37,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:39,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:44,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:47,972 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:50,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:50,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:35:50,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:35:50,675 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:35:51,534 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:35:55,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:58,028 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:01,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:06,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:36:06,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:36:06,050 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:36:06,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:08,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:11,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:16,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:18,141 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:21,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:36:21,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:36:21,049 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:36:21,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:21,533 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:36:26,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:31,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:32,223 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:36,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:36:36,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:36:36,050 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:36:37,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:42,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:44,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:47,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:51,535 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:36:52,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:36:52,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:36:52,352 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:36:53,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:54,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:58,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:03,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:04,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:07,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:37:07,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:37:07,351 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:37:08,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:12,484 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:14,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:19,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:21,535 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:37:22,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:37:22,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:37:22,352 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:37:22,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:24,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:29,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:33,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:37:33,214 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:37:33,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:37:33,217 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:37:33,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:37:34,623 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:35,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:35,629 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:37,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:37:37,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:37:37,351 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:37:40,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:45,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:45,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:50,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:51,536 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:37:52,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:37:52,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:37:52,410 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:37:53,749 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:56,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:01,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:03,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:07,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:07,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:38:07,409 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:38:11,863 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:13,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:18,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:21,536 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:38:21,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:22,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:38:22,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:38:22,411 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:38:23,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:28,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:33,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:33,995 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:37,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:38:37,409 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:38:39,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:44,065 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:44,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:49,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:51,537 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:38:52,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:38:52,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:38:52,544 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:38:55,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:56,132 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:00,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:04,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:05,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:07,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:39:07,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:39:11,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:14,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:16,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:21,538 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:39:22,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:22,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:39:22,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:39:26,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:28,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:33,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:36,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:37,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:39:37,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:39:38,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:43,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:48,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:48,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:51,538 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:39:52,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:39:52,543 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:39:54,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:58,518 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:00,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:05,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:07,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:40:07,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:40:07,596 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:40:08,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:10,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:15,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:18,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:20,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:21,538 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:40:22,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:40:22,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:40:26,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:31,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:32,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:36,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:37,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:40:37,543 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:40:42,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:42,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:47,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:51,539 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:40:52,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:52,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:40:52,545 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:40:52,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:57,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:02,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:41:02,143 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:41:02,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:41:02,146 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:41:02,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:41:03,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:03,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:07,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:41:07,544 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:41:09,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:14,036 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:15,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:20,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:21,541 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:41:22,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:41:22,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:41:25,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:26,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:30,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:35,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:37,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:41:37,980 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:41:38,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:40,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:45,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:46,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:50,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:51,541 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:41:54,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:41:54,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:41:54,658 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:41:55,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:58,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:00,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:05,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:08,361 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:09,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:42:09,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:42:09,656 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:42:10,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:15,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:21,541 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:42:21,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:22,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:24,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:42:24,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:42:24,657 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:42:26,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:32,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:34,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:37,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:39,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:42:39,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:42:39,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:42:42,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:43,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:48,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:51,542 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:42:53,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:54,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:54,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:42:54,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:42:54,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:42:59,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:02,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:04,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:09,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:09,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:43:09,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:43:09,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:43:12,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:15,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:21,011 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:21,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:21,542 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:43:24,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:43:24,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:43:24,891 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:43:26,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:30,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:31,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:36,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:39,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:43:39,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:43:39,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:43:42,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:42,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:47,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:51,543 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:43:53,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:54,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:54,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:43:54,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:43:54,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:43:58,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:02,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:03,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:08,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:09,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:44:09,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:44:09,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:44:14,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:16,362 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:20,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:21,544 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:44:24,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:44:24,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:44:24,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:44:25,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:26,419 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:30,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:35,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:37,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:44:37,036 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:44:37,037 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:44:37,039 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:44:37,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:44:38,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:39,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:39,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:44:39,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:44:39,891 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:44:40,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:45,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:50,559 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:51,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:51,544 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:44:54,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:44:54,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:44:54,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:44:56,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:00,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:02,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:07,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:09,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:45:09,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:45:09,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:45:10,684 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:12,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:17,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:21,545 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:45:22,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:22,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:24,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:45:24,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:45:24,891 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:45:28,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:33,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:34,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:38,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:40,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:45:40,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:45:40,892 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:45:44,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:44,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:49,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:51,546 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:45:54,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:54,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:55,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:45:55,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:00,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:05,044 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:05,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:10,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:10,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:46:10,892 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:16,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:17,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:21,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:21,546 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:46:25,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:46:25,891 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:27,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:27,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:32,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:35,241 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:37,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:41,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:46:41,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:46:41,276 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:43,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:45,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:48,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:51,546 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:46:52,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:53,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:56,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:46:56,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:46:56,275 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:59,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:04,423 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:04,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:09,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:12,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:47:12,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:47:12,813 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:47:14,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:15,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:20,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:21,548 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:47:24,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:26,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:27,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:47:27,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:47:27,811 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:47:32,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:32,627 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:37,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:42,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:42,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:47:42,812 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:47:42,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:47:43,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:48,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:51,548 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:47:52,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:53,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:57,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:47:57,812 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:47:58,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:03,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:48:03,365 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:48:03,366 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:48:03,369 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:48:03,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:48:04,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:04,824 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:05,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:09,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:13,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:48:13,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:48:13,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:48:14,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:16,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:19,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:21,550 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:48:24,979 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:25,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:28,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:48:28,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:48:28,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:48:30,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:35,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:36,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:41,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:43,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:48:43,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:48:43,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:48:45,117 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:46,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:51,550 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:48:51,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:53,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:56,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:58,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:48:58,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:48:58,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:49:02,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:05,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:08,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:13,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:49:13,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:49:13,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:49:13,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:15,310 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:18,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:21,550 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:49:23,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:24,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:29,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:49:29,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:49:29,012 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:49:29,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:31,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:34,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:39,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:43,505 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:44,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:49:44,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:49:44,011 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:49:45,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:50,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:51,551 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:49:52,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:55,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:59,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:49:59,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:00,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:02,627 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:05,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:10,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:14,690 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:14,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:50:14,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:15,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:20,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:21,551 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:50:22,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:26,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:29,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:50:29,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:50:29,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:31,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:37,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:38,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:42,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:44,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:50:44,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:50:44,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:46,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:48,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:51,551 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:50:53,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:58,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:58,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:59,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:50:59,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:59,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:51:03,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:09,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:11,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:14,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:51:14,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:51:14,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:19,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:20,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:21,552 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:51:26,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:29,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:51:29,761 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:51:30,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:51:30,832 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:51:30,832 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:51:30,835 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:51:31,174 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:51:31,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:32,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:33,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:36,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:41,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:43,251 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:44,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:51:44,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:51:46,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:51,553 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:51:52,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:55,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:58,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:59,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:51:59,760 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:52:04,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:05,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:09,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:13,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:14,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:52:14,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:52:15,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:20,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:21,489 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:21,553 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:52:26,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:29,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:52:29,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:52:31,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:31,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:36,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:41,596 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:42,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:44,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:52:44,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:52:47,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:51,554 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:52:53,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:53,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:58,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:00,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:53:00,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:53:00,786 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:53:03,726 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:03,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:08,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:13,787 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:14,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:15,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:53:15,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:53:15,785 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:53:19,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:21,555 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:53:24,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:25,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:30,503 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:30,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:53:30,785 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:53:32,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:36,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:41,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:42,963 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:45,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:53:45,784 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:53:47,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:51,024 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:51,555 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:53:52,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:57,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:00,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:54:00,786 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:54:01,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:02,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:07,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:11,148 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:13,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:15,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:54:15,784 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:54:18,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:21,207 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:21,556 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:54:24,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:29,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:30,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:54:30,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:54:30,851 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:54:33,281 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:35,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:40,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:45,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:45,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:45,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:54:45,849 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:54:51,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:51,556 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:54:54,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:54:54,940 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:54:54,941 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:54:54,944 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:54:55,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:54:56,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:56,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:57,423 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:00,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:55:00,849 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:55:02,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:07,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:07,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:12,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:16,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:55:16,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:55:16,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:55:18,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:19,554 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:21,557 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:55:23,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:28,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:31,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:55:31,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:55:33,665 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:33,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:38,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:41,719 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:44,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:46,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:55:46,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:55:46,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:55:49,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:51,557 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:55:51,783 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:55,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:59,836 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:00,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:01,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:56:01,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:56:01,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:56:06,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:11,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:11,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:16,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:56:16,138 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:56:17,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:21,558 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:56:21,959 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:22,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:27,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:32,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:56:32,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:56:32,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:56:33,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:35,119 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:39,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:44,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:47,187 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:47,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:56:47,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:56:47,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:56:49,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:51,561 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:56:55,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:55,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:00,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:02,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:57:02,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:57:02,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:57:05,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:07,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:10,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:15,379 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:16,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:17,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:57:17,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:57:21,562 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:57:21,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:25,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:27,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:32,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:32,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:57:32,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:57:32,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:57:35,509 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:37,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:43,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:43,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:47,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:57:47,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:57:47,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:57:48,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:51,562 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:57:54,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:55,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:59,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:02,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:58:02,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:58:02,660 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:58:03,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:04,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:09,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:13,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:15,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:17,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:58:17,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:58:17,661 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:58:20,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:21,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:58:21,450 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:58:21,450 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:58:21,453 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:58:21,563 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:58:21,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:58:22,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:23,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:26,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:31,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:32,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:32,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:58:32,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:58:32,661 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:58:37,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:43,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:43,956 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:47,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:58:47,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:58:47,664 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:58:48,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:51,563 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:58:52,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:54,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:59,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:00,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:02,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:59:02,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:59:02,661 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:59:04,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:07,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:08,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:10,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:15,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:16,167 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:17,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:59:17,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:59:17,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:59:20,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:21,564 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:59:26,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:31,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:32,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:32,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:59:32,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:59:32,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:59:37,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:41,311 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:42,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:47,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:59:47,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:59:47,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:59:48,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:49,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:51,564 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:59:54,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:59,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:01,434 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:02,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:00:02,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:00:02,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:00:04,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:09,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:13,507 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:15,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:17,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:00:17,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:00:17,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:00:20,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:21,565 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:00:23,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:26,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:31,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:32,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:00:32,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:00:32,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:00:33,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:36,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:41,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:45,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:47,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:47,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:00:47,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:00:47,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:00:51,566 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:00:52,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:57,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:57,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:02,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:01:02,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:01:02,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:01:02,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:07,845 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:08,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:13,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:15,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:17,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:01:17,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:01:17,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:01:18,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:21,566 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:01:24,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:25,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:29,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:32,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:01:32,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:01:32,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:01:34,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:34,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:39,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:43,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:01:43,791 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:01:43,792 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:01:43,794 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:01:44,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:01:45,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:45,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:46,104 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:47,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:01:47,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:01:47,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:01:50,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:51,567 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:01:54,153 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:56,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:01,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:02,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:02:02,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:02:02,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:02:04,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:06,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:12,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:14,288 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:17,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:17,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:02:17,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:02:17,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:02:21,567 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:02:22,341 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:22,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:27,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:32,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:32,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:02:32,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:02:32,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:02:34,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:38,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:43,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:44,477 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:47,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:02:47,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:02:47,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:02:48,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:51,568 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:02:54,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:55,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:59,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:02,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:03:02,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:03:02,670 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:03:03,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:05,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:10,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:13,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:15,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:17,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:03:17,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:03:17,671 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:03:21,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:21,569 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:03:21,729 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:26,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:31,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:32,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:32,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:03:32,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:03:32,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:03:38,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:43,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:44,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:47,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:03:47,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:03:47,670 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:03:49,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:51,569 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:03:51,931 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:55,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:01,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:01,999 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:02,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:04:02,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:04:02,670 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:04:06,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:10,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:12,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:17,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:17,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:04:17,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:04:17,691 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:04:20,108 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:21,569 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:04:22,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:27,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:30,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:32,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:04:32,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:04:32,675 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:04:32,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:38,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:42,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:44,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:48,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:04:48,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:04:48,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:04:49,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:51,570 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:04:52,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:55,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:58,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:04:58,336 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:04:58,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:04:58,338 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:04:58,347 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:05:00,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:01,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:01,366 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:03,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:05:03,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:05:03,221 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:05:06,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:08,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:11,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:16,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:17,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:18,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:05:18,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:05:18,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:05:21,570 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:05:22,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:28,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:30,567 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:33,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:05:33,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:05:33,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:05:33,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:38,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:42,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:44,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:49,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:05:49,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:05:49,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:05:49,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:51,571 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:05:52,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:55,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:00,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:03,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:04,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:06:04,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:06:04,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:06:05,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:10,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:15,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:16,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:19,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:06:19,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:06:19,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:06:21,573 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:06:21,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:23,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:27,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:32,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:33,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:34,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:06:34,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:06:38,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:43,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:44,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:48,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:50,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:06:50,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:06:50,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:06:51,573 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:06:53,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:54,088 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:58,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:03,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:04,149 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:07:05,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:07:05,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:07:09,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:14,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:07:14,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:19,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:20,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:07:20,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:07:21,574 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:07:24,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:26,280 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:07:29,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:35,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:07:35,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:07:35,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:36,336 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:07:41,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:46,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:50,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:07:50,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:07:51,574 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:07:51,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:57,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:58,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:02,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:05,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:08:05,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:08:06,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:07,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:12,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:16,562 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:17,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:20,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:08:20,270 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:08:21,576 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:08:23,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:24,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:28,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:32,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:08:32,176 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:08:32,176 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:08:32,179 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:08:32,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:08:33,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:33,682 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:34,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:35,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:08:35,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:08:38,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:44,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:44,758 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:49,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:50,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:08:50,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:08:50,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:08:51,576 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:08:54,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:54,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:59,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:02,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:05,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:05,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:09:05,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:09:10,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:12,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:15,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:20,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:09:20,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:09:20,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:21,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:21,577 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:09:26,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:30,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:32,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:37,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:09:37,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:09:37,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:09:38,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:40,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:43,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:48,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:49,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:51,577 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:09:52,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:09:52,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:09:52,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:09:54,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:59,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:00,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:04,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:07,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:10:07,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:10:07,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:10:09,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:10,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:14,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:20,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:20,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:21,577 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:10:22,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:10:22,095 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:10:26,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:30,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:31,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:36,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:37,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:10:37,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:10:37,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:10:40,493 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:42,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:47,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:48,555 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:51,578 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:10:52,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:10:52,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:10:52,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:58,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:58,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:03,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:06,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:07,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:11:07,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:11:09,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:14,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:18,745 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:20,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:21,579 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:11:22,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:11:22,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:11:22,126 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:11:26,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:26,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:31,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:36,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:36,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:37,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:11:37,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:11:42,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:42,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:11:42,671 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:11:42,671 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:11:42,673 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:11:42,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:11:44,921 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:45,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:47,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:51,579 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:11:52,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:11:52,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:11:52,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:53,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:58,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:03,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:03,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:07,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:12:07,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:12:09,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:15,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:15,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:20,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:21,580 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:12:22,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:12:22,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:12:23,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:25,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:30,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:33,260 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:35,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:38,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:12:38,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:12:38,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:12:40,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:40,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:45,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:50,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:51,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:51,580 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:12:53,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:12:53,374 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:12:56,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:58,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:01,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:06,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:08,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:13:08,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:13:08,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:13:10,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:12,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:17,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:18,556 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:21,581 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:13:23,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:13:23,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:23,375 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:13:28,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:28,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:33,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:36,804 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:38,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:13:38,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:13:38,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:13:39,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:44,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:48,885 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:50,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:51,581 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:13:53,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:13:53,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:13:55,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:58,944 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:01,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:06,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:08,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:14:08,375 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:14:09,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:11,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:16,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:21,068 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:21,582 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:14:21,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:23,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:14:23,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:14:26,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:31,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:32,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:37,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:38,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:14:38,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:14:41,204 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:42,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:47,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:51,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:51,582 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:14:53,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:53,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:14:53,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:14:58,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:14:58,257 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:14:58,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:14:58,260 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:14:58,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:14:59,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:59,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:01,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:04,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:08,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:15:08,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:15:09,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:13,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:15,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:20,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:21,582 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:15:22,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:23,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:15:23,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:15:25,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:30,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:31,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:36,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:38,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:15:38,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:15:38,382 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:15:40,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:41,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:46,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:50,661 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:51,583 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:15:52,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:53,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:15:53,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:15:53,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:15:58,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:00,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:03,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:08,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:16:08,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:16:08,382 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:16:09,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:10,801 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:14,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:18,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:19,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:21,583 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:16:23,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:16:23,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:16:23,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:16:25,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:30,932 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:31,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:36,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:38,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:16:38,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:16:38,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:16:38,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:41,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:47,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:49,041 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:51,584 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:16:53,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:16:53,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:16:53,382 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:16:53,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:58,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:59,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:03,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:07,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:08,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:17:08,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:17:08,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:17:09,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:14,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:17,229 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:19,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:21,584 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:17:23,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:17:23,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:17:23,383 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:17:25,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:29,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:31,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:36,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:37,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:38,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:17:38,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:17:38,382 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:17:42,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:47,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:47,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:51,585 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:17:53,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:53,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:17:53,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:17:53,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:17:57,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:58,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:04,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:05,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:08,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:18:08,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:18:10,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:12,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:18:12,783 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:18:12,784 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:18:12,787 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:18:13,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:18:13,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:15,582 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:15,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:20,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:21,586 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:18:23,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:18:23,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:18:23,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:18:25,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:26,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:31,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:35,697 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:36,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:38,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:18:38,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:18:42,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:46,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:47,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:51,587 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:18:53,590 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:53,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:18:53,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:18:54,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:59,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:04,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:08,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:19:08,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:19:10,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:15,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:20,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:21,587 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:19:23,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:19:23,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:19:25,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:30,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:35,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:38,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:19:38,925 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:19:40,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:41,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:19:45,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:49,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:19:50,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:51,588 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:19:53,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:19:53,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:19:56,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:59,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:01,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:06,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:08,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:20:08,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:20:09,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:12,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:17,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:19,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:21,588 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:20:23,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:24,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:20:24,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:20:24,509 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:20:27,365 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:28,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:33,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:37,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:39,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:39,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:20:39,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:20:44,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:47,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:49,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:51,589 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:20:54,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:20:54,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:20:54,554 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:20:54,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:57,560 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:59,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:04,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:07,651 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:09,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:21:09,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:21:10,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:15,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:16,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:21,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:21,591 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:21:24,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:21:24,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:21:25,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:26,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:31,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:33,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:37,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:39,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:21:39,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:21:42,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:45,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:47,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:51,591 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:21:53,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:53,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:54,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:21:54,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:21:59,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:00,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:22:00,986 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:22:00,986 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:22:00,988 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:22:01,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:22:02,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:03,357 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:04,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:09,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:22:09,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:22:09,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:22:10,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:11,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:15,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:20,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:21,592 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:22:24,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:22:24,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:22:26,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:31,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:36,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:39,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:22:39,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:22:41,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:45,596 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:47,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:51,593 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:22:53,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:53,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:54,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:22:54,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:22:58,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:03,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:05,744 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:08,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:10,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:23:10,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:23:10,847 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:23:13,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:14,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:19,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:21,593 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:23:23,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:24,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:25,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:23:25,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:23:30,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:31,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:35,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:40,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:40,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:23:40,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:23:41,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:46,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:51,593 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:23:51,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:52,036 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:55,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:23:55,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:23:55,848 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:23:57,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:00,091 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:02,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:07,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:09,443 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:10,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:24:10,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:24:13,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:17,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:18,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:21,594 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:24:24,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:25,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:24:25,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:24:27,554 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:30,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:35,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:37,621 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:40,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:40,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:24:40,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:24:46,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:51,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:51,594 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:24:51,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:56,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:56,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:24:56,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:24:56,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:24:59,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:02,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:07,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:09,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:11,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:25:11,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:25:11,929 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:25:12,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:17,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:17,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:21,606 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:25:22,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:26,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:25:26,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:25:26,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:25:27,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:29,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:32,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:36,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:25:36,799 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:25:36,800 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:25:36,802 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:25:36,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:25:37,997 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:38,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:41,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:25:41,929 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:25:44,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:48,104 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:49,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:51,595 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:25:55,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:56,166 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:56,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:25:56,930 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:26:01,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:06,230 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:07,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:11,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:26:11,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:26:11,962 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:26:12,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:15,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:18,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:21,597 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:26:23,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:25,358 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:26,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:26:26,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:26:29,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:33,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:34,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:39,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:41,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:26:41,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:26:43,481 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:45,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:50,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:51,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:51,597 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:26:55,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:58,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:26:58,230 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:27:01,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:01,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:06,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:09,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:11,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:13,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:27:13,229 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:27:17,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:19,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:21,597 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:27:22,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:28,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:27:28,229 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:27:28,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:29,784 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:33,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:37,845 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:38,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:43,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:44,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:27:44,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:27:44,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:27:47,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:48,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:51,598 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:27:54,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:55,967 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:59,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:59,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:27:59,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:27:59,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:28:04,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:08,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:10,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:14,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:28:14,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:28:14,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:28:15,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:16,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:20,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:21,599 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:28:24,150 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:26,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:30,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:28:30,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:28:30,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:28:32,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:34,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:37,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:41,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:28:41,530 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:28:41,531 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:28:41,534 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:28:42,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:28:42,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:42,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:44,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:45,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:28:45,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:28:45,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:28:48,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:51,600 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:28:54,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:54,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:59,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:00,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:29:00,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:29:00,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:29:02,393 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:04,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:09,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:12,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:14,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:15,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:29:15,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:29:15,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:29:20,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:21,601 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:29:25,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:25,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:30,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:30,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:29:30,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:29:30,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:29:35,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:35,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:40,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:45,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:45,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:45,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:29:45,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:29:45,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:29:51,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:51,602 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:29:53,730 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:56,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:00,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:30:00,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:30:00,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:30:02,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:03,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:07,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:13,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:13,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:15,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:30:15,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:30:15,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:30:19,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:21,602 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:30:21,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:24,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:29,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:30,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:30:30,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:30:30,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:30:31,994 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:35,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:40,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:40,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:45,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:47,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:30:47,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:30:47,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:30:50,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:50,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:51,603 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:30:55,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:00,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:00,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:02,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:31:02,321 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:31:06,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:10,329 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:12,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:17,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:17,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:31:17,321 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:31:20,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:21,603 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:31:22,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:28,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:28,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:32,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:31:32,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:31:32,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:31:33,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:36,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:39,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:44,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:46,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:47,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:31:47,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:31:49,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:51,605 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:31:53,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:31:53,526 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:31:53,527 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:31:53,529 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:31:53,628 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:31:54,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:55,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:00,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:03,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:32:03,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:32:03,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:32:06,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:06,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:11,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:13,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:16,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:18,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:32:18,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:32:21,606 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:32:22,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:23,831 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:27,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:33,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:33,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:32:33,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:32:33,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:38,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:43,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:48,065 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:48,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:32:48,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:32:48,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:51,606 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:32:53,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:56,124 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:59,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:03,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:33:03,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:33:03,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:33:04,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:06,187 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:09,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:14,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:14,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:18,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:33:18,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:33:20,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:21,608 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:33:24,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:25,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:30,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:33,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:33:33,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:33:34,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:36,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:41,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:44,451 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:47,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:48,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:33:48,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:33:48,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:33:51,608 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:33:52,515 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:52,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:57,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:02,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:03,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:34:03,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:34:04,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:08,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:13,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:14,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:18,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:34:18,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:34:19,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:21,609 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:34:24,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:24,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:29,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:33,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:34:33,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:34:34,800 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:35,590 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:40,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:44,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:46,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:48,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:34:48,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:34:48,340 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:34:51,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:51,609 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:34:52,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:56,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:02,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:02,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:03,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:35:03,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:35:07,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:12,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:13,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:35:13,644 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:35:13,645 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:35:13,647 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:35:14,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:35:14,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:15,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:17,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:18,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:35:18,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:35:21,610 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:35:22,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:24,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:27,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:32,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:33,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:35:33,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:35:34,162 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:38,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:42,218 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:44,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:48,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:35:48,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:35:48,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:35:50,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:51,611 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:35:52,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:55,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:01,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:02,353 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:03,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:36:03,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:36:03,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:36:07,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:10,409 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:12,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:17,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:18,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:36:18,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:36:18,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:36:20,475 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:21,611 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:36:22,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:27,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:28,528 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:32,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:33,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:36:33,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:36:33,925 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:36:38,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:38,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:43,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:46,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:48,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:36:48,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:36:49,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:51,612 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:36:54,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:58,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:00,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:03,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:37:03,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:37:03,998 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:37:06,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:06,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:11,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:14,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:16,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:18,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:37:18,996 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:37:21,614 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:37:22,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:24,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:27,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:33,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:33,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:37:33,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:37:34,979 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:39,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:44,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:45,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:49,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:50,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:37:50,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:37:50,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:37:51,614 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:37:53,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:54,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:59,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:03,162 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:05,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:38:05,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:38:05,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:05,362 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:38:10,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:10,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:15,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:18,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:38:18,773 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:38:18,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:38:18,776 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:38:19,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:38:20,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:20,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:38:20,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:38:20,362 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:38:21,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:21,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:21,614 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:38:26,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:28,358 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:31,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:35,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:38:35,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:38:35,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:38:37,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:40,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:42,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:47,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:48,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:50,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:38:50,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:38:50,362 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:38:51,631 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:38:53,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:58,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:59,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:04,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:05,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:39:05,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:39:05,365 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:39:08,611 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:09,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:14,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:16,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:19,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:20,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:39:20,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:39:20,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:39:21,631 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:39:24,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:26,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:29,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:34,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:35,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:39:35,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:39:35,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:39:36,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:40,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:45,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:46,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:50,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:39:50,364 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:39:51,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:51,633 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:39:54,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:56,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:01,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:05,038 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:05,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:40:05,364 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:40:07,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:12,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:15,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:18,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:20,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:40:21,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:40:21,002 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:40:21,634 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:40:23,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:23,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:28,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:32,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:34,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:36,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:40:36,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:40:36,001 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:40:39,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:40,484 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:45,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:50,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:50,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:51,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:40:51,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:40:51,001 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:40:51,634 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:40:55,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:58,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:01,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:06,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:07,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:41:07,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:41:07,259 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:41:08,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:11,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:16,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:18,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:21,634 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:41:21,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:22,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:41:22,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:41:27,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:27,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:41:27,056 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:41:27,056 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:41:27,059 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:41:27,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:41:28,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:29,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:32,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:37,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:37,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:41:37,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:41:38,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:42,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:47,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:48,948 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:51,635 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:41:52,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:41:52,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:41:52,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:58,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:59,015 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:03,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:07,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:42:07,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:42:08,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:09,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:13,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:17,150 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:18,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:21,635 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:42:22,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:42:22,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:42:22,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:42:24,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:27,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:29,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:35,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:35,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:37,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:42:37,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:42:40,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:46,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:47,451 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:51,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:51,637 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:42:53,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:42:53,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:42:53,044 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:42:55,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:56,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:01,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:05,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:06,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:08,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:43:08,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:43:08,041 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:43:12,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:18,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:18,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:21,637 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:43:23,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:43:23,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:43:23,041 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:43:23,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:28,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:28,791 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:33,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:38,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:43:38,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:43:38,043 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:43:38,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:40,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:43,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:48,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:49,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:51,638 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:43:53,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:43:53,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:43:53,042 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:43:55,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:58,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:00,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:05,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:08,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:44:08,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:44:08,042 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:44:09,060 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:11,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:16,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:17,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:21,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:21,638 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:44:24,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:44:24,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:44:24,340 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:44:27,300 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:27,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:32,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:37,357 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:38,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:39,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:44:39,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:44:43,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:45,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:44:45,567 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:44:45,567 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:44:45,568 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:44:46,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:44:47,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:48,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:51,640 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:44:53,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:54,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:44:54,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:44:54,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:44:55,492 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:59,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:04,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:05,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:09,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:45:09,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:45:10,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:13,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:15,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:20,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:21,643 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:45:24,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:45:24,340 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:45:25,695 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:26,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:31,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:35,764 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:36,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:40,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:45:40,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:45:40,855 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:45:41,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:43,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:46,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:51,641 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:45:52,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:52,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:55,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:45:55,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:45:55,855 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:45:58,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:00,938 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:04,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:09,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:10,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:46:10,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:46:11,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:14,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:19,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:21,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:21,642 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:46:24,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:25,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:46:25,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:46:25,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:46:29,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:31,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:34,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:39,202 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:40,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:46:40,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:46:41,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:46,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:49,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:51,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:51,642 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:46:55,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:46:55,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:46:55,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:46:56,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:57,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:47:01,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:06,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:07,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:47:10,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:47:10,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:47:10,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:47:12,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:17,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:21,643 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:47:22,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:25,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:47:25,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:47:25,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:47:28,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:33,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:38,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:40,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:47:40,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:47:40,896 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:47:44,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:49,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:51,643 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:47:54,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:55,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:47:55,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:47:55,896 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:00,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:03,930 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:05,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:10,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:48:10,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:10,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:48:10,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:11,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:16,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:21,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:21,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:21,644 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:48:25,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:48:25,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:48:25,859 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:27,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:29,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:32,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:37,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:39,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:48:39,858 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:48:39,858 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:48:39,861 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:48:40,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:48:40,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:48:40,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:48:40,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:41,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:42,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:43,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:48,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:51,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:51,644 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:48:53,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:57,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:48:57,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:48:57,088 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:59,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:59,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:04,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:09,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:10,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:12,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:49:12,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:49:12,086 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:49:15,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:17,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:20,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:21,645 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:49:25,465 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:25,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:27,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:49:27,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:49:27,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:49:31,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:36,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:37,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:41,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:43,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:49:43,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:49:43,016 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:49:45,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:47,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:51,646 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:49:52,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:55,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:58,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:58,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:49:58,016 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:03,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:04,029 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:08,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:13,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:50:13,015 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:13,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:14,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:18,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:21,647 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:50:23,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:24,157 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:28,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:29,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:50:29,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:50:29,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:32,213 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:34,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:39,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:41,266 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:44,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:50:44,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:50:44,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:44,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:49,322 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:49,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:51,647 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:50:55,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:59,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:50:59,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:50:59,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:59,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:01,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:06,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:11,459 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:12,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:14,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:51:14,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:51:14,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:51:17,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:21,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:21,648 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:51:22,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:27,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:29,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:51:29,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:51:29,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:51:29,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:33,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:39,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:39,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:44,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:44,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:51:44,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:51:44,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:51:47,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:51:47,396 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:51:47,397 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:51:47,399 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:51:48,003 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:51:49,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:50,015 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:51,648 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:51:54,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:59,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:51:59,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:51:59,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:52:00,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:00,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:05,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:08,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:10,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:14,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:52:14,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:52:14,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:52:16,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:18,202 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:21,650 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:52:22,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:26,255 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:27,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:29,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:52:29,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:52:29,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:52:32,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:36,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:37,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:42,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:44,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:52:44,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:52:46,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:48,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:51,651 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:52:53,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:55,452 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:59,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:59,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:52:59,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:53:04,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:05,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:09,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:13,704 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:14,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:14,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:53:14,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:53:20,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:21,651 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:53:23,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:25,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:30,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:31,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:53:31,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:53:31,066 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:53:33,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:36,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:42,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:43,888 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:46,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:53:46,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:53:46,065 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:53:47,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:51,652 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:53:51,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:52,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:57,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:01,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:54:01,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:54:01,065 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:54:02,050 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:03,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:09,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:10,139 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:14,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:16,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:54:16,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:54:16,084 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:54:19,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:20,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:21,652 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:54:24,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:29,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:31,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:54:31,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:54:31,085 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:54:34,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:35,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:40,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:42,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:45,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:46,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:54:46,082 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:54:50,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:51,652 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:54:52,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:55,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:00,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:01,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:55:01,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:55:06,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:11,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:14,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:55:14,635 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:55:14,636 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:55:14,638 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:55:15,609 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:55:16,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:55:16,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:55:16,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:16,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:21,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:21,653 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:55:26,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:29,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:31,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:55:31,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:55:32,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:37,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:38,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:43,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:46,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:55:46,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:55:47,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:48,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:51,654 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:55:53,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:55,864 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:58,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:01,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:56:01,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:56:01,085 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:56:04,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:05,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:09,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:13,984 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:15,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:16,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:56:16,084 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:56:21,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:21,655 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:56:24,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:27,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:31,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:56:31,085 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:56:33,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:34,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:38,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:42,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:44,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:46,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:56:46,084 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:56:49,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:51,655 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:56:52,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:54,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:59,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:00,440 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:01,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:57:01,084 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:57:05,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:10,511 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:10,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:15,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:16,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:57:16,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:57:16,580 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:57:20,576 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:20,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:21,655 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:57:26,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:28,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:31,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:57:31,578 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:57:31,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:37,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:38,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:42,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:46,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:57:46,578 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:57:46,758 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:47,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:51,656 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:57:53,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:56,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:58,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:01,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:58:01,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:58:01,595 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:58:03,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:04,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:09,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:13,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:15,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:16,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:58:16,593 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:58:20,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:21,657 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:58:26,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:26,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:58:26,646 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:58:26,647 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:58:26,649 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:58:27,026 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:58:28,030 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:31,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:58:31,593 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:58:31,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:37,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:38,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:42,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:46,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:46,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:58:46,593 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:58:47,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:51,659 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:58:53,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:54,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:58,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:01,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:59:01,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:59:01,596 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:59:04,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:04,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:09,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:12,342 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:14,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:16,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:59:16,595 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:59:19,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:21,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:59:24,414 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:24,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:29,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:31,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:59:31,595 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:59:32,467 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:35,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:41,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:42,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:46,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:47,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:59:47,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:59:47,978 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:59:50,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:51,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:51,659 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:59:56,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:00,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:01,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:02,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:00:02,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:00:02,975 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:00:07,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:10,713 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:13,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:17,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:00:17,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:00:17,974 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:00:18,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:20,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:21,660 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:00:23,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:28,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:28,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:32,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:00:32,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:00:32,977 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:00:34,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:38,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:39,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:45,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:46,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:47,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:00:47,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:00:47,976 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:00:51,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:51,660 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:00:57,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:57,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:02,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:02,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:01:02,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:01:02,978 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:01:07,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:08,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:12,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:17,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:17,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:01:17,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:01:17,976 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:01:18,182 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:21,661 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:01:23,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:28,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:29,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:32,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:01:32,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:01:32,980 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:01:34,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:36,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:01:36,086 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:01:36,086 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:01:36,088 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:01:36,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:01:38,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:40,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:46,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:46,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:47,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:01:47,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:01:47,977 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:01:51,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:51,662 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:01:56,448 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:57,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:02,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:02,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:02:02,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:02:02,979 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:02:07,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:08,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:12,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:17,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:02:17,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:17,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:02:17,977 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:02:18,578 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:21,662 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:02:23,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:26,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:29,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:33,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:02:33,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:02:33,196 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:02:35,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:36,693 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:40,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:44,758 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:46,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:48,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:02:48,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:02:48,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:02:51,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:51,663 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:02:54,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:57,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:02,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:03,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:03:03,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:03:03,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:03:07,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:12,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:14,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:17,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:18,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:03:18,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:03:18,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:03:21,664 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:03:22,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:23,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:27,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:32,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:33,072 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:33,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:03:33,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:03:33,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:03:38,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:41,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:43,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:48,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:03:48,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:03:48,195 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:03:49,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:51,188 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:51,664 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:03:54,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:00,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:00,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:03,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:04:03,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:04:03,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:04:05,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:10,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:10,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:15,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:18,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:04:18,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:04:18,195 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:04:18,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:21,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:21,664 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:04:26,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:28,705 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:31,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:34,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:04:34,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:04:34,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:04:36,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:36,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:41,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:46,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:46,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:49,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:04:49,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:04:49,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:04:51,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:51,666 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:04:53,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:04:53,446 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:04:53,446 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:04:53,448 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:04:53,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:04:54,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:57,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:02,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:04,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:05:04,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:05:04,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:05:07,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:07,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:12,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:15,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:18,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:19,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:05:19,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:05:19,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:05:21,666 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:05:23,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:25,131 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:28,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:33,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:33,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:34,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:05:34,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:05:38,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:41,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:43,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:48,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:49,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:05:49,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:05:49,358 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:05:51,667 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:05:53,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:54,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:00,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:01,385 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:04,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:06:04,357 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:06:05,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:11,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:11,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:16,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:18,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:19,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:06:19,357 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:06:21,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:21,667 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:06:26,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:27,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:32,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:34,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:06:34,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:06:34,365 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:06:36,610 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:37,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:42,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:46,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:47,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:49,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:06:49,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:06:51,667 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:06:53,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:56,732 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:59,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:04,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:07:04,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:07:04,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:04,785 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:09,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:14,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:14,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:19,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:20,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:07:20,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:07:20,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:07:21,668 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:07:22,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:25,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:30,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:32,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:35,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:07:35,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:07:35,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:40,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:43,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:46,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:50,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:07:50,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:07:51,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:51,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:07:51,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:56,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:59,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:07:59,137 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:07:59,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:07:59,139 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:07:59,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:08:01,206 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:08:02,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:03,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:08:05,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:08:05,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:08:05,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:08:07,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:12,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:17,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:20,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:08:20,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:08:21,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:08:23,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:28,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:33,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:35,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:08:35,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:08:38,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:43,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:48,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:50,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:08:50,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:08:51,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:08:54,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:59,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:04,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:05,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:09:05,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:09:09,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:14,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:19,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:20,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:09:20,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:09:21,670 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:09:25,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:30,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:35,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:09:35,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:09:35,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:40,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:45,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:50,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:09:50,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:09:51,671 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:09:51,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:56,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:01,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:05,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:10:05,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:10:06,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:11,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:16,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:20,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:10:20,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:10:21,671 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:10:22,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:27,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:32,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:35,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:10:35,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:10:37,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:42,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:47,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:50,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:10:50,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:10:51,671 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:10:53,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:58,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:03,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:05,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:11:05,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:11:08,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:13,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:18,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:20,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:11:20,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:11:20,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:11:21,672 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:11:24,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:29,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:34,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:35,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:11:35,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:11:39,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:44,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:49,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:50,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:11:50,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:11:51,672 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:11:55,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:00,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:05,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:12:05,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:12:05,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:10,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:15,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:20,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:12:20,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:12:21,673 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:12:21,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:26,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:31,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:35,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:12:35,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:12:36,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:41,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:46,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:50,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:12:50,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:12:51,674 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:12:52,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:57,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:02,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:05,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:13:05,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:13:07,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:12,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:17,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:20,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:13:20,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:13:21,674 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:13:23,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:28,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:33,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:35,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:13:35,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:13:38,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:43,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:48,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:50,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:13:50,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:13:51,674 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:13:54,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:59,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:04,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:05,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:14:05,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:14:09,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:14,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:19,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:20,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:14:20,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:14:21,675 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:14:25,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:30,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:35,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:14:35,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:14:35,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:40,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:45,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:50,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:14:50,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:14:50,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:51,675 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:14:56,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:01,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:05,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:15:05,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:15:06,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:11,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:16,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:20,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:15:20,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:15:21,675 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:15:22,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:27,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:32,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:35,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:15:35,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:15:37,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:42,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:47,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:50,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:15:50,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:15:51,676 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:15:53,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:58,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:03,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:05,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:16:05,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:16:08,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:13,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:18,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:20,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:16:20,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:16:21,676 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:16:24,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:29,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:34,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:35,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:16:35,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:16:39,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:44,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:49,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:50,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:16:50,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:16:51,676 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:16:55,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:00,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:05,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:17:05,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:17:05,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:10,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:15,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:20,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:17:20,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:17:21,677 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:17:21,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:26,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:31,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:35,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:17:35,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:17:36,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:41,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:46,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:50,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:17:50,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:17:51,677 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:17:52,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:57,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:02,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:05,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:18:05,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:18:07,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:11,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:18:11,252 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,252 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,253 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,253 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,253 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,263 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:18:11,263 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:18:11,267 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:18:11,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:18:12,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:13,949 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:18:17,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:18,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/config.yaml 2023-11-18 16:18:20,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:18:20,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:18:20,652 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:18:21,678 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:18:23,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:28,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:33,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:36,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:18:36,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:18:36,820 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:18:38,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:44,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:18:44,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:46,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:18:49,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:51,687 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:18:55,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:57,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:19:00,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:05,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:10,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:15,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:20,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:21,679 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:19:26,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:31,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:36,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:41,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:46,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:46,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:19:46,825 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:19:51,683 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:19:52,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:53,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:19:54,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:19:54,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:19:55,381 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:19:57,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:19:58,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:03,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:08,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:09,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:20:13,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:15,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:20:15,810 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:20:17,486 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:20:19,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:21,683 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:20:24,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:24,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:20:24,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:20:24,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:20:29,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:35,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:39,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:20:39,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:20:40,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:46,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:51,683 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:20:52,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:54,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:20:54,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:20:54,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:20:57,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:03,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:08,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:09,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:21:09,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:21:09,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:21:13,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:13,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:19,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:21,705 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:21:23,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:24,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:21:24,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:21:26,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:31,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:35,845 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:36,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:39,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:21:39,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:21:39,745 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:21:42,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:45,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:47,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:51,710 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:21:53,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:54,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:21:54,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:21:55,999 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:58,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:04,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:06,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:09,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:22:09,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:22:09,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:14,297 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:15,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:20,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:21,711 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:22:24,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:24,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:22:24,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:22:25,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:31,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:36,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:38,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:39,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:22:39,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:22:41,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:46,561 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:47,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:51,712 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:22:53,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:54,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:22:54,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:22:56,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:58,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:04,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:05,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:09,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:09,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:23:09,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:23:09,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:23:13,751 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:15,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:21,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:21,714 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:23:23,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:24,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:23:24,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:23:26,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:31,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:33,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:37,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:39,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:23:39,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:23:39,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:23:43,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:44,081 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:49,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:51,714 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:23:54,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:54,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:54,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:23:54,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:23:59,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:02,206 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:05,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:09,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:24:09,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:24:10,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:12,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:15,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:20,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:21,714 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:24:22,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:24,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:24:24,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:24:25,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:30,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:31,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:24:31,149 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:24:31,149 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:24:31,152 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:24:31,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:24:32,407 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:36,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:39,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:24:39,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:24:41,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:42,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:46,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:51,715 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:24:52,520 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:52,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:54,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:24:54,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:24:57,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:00,583 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:03,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:09,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:09,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:25:09,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:25:10,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:15,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:18,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:20,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:21,715 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:25:25,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:25:25,179 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:25:26,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:28,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:31,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:36,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:40,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:25:40,180 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:25:40,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:25:42,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:42,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:47,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:51,717 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:25:51,912 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:52,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:55,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:25:55,179 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:25:58,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:03,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:05,996 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:08,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:10,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:26:10,179 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:26:13,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:14,054 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:18,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:21,717 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:26:24,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:24,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:25,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:26:25,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:26:30,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:34,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:36,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:41,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:41,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:26:41,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:26:41,567 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:26:44,256 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:46,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:51,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:51,717 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:26:52,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:56,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:26:56,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:26:56,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:26:56,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:01,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:02,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:07,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:11,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:27:11,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:27:11,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:27:12,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:12,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:17,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:20,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:21,717 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:27:23,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:26,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:27:26,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:27:26,578 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:27:28,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:30,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:34,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:39,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:40,699 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:41,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:27:41,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:27:41,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:27:45,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:47,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:27:47,828 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:27:47,829 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:27:47,831 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:27:48,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:27:48,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:50,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:51,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:51,718 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:27:56,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:27:56,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:27:56,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:27:56,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:58,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:01,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:06,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:07,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:11,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:28:11,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:28:11,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:28:12,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:17,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:19,017 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:21,718 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:28:23,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:26,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:28:26,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:28:26,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:28:27,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:28,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:34,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:34,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:39,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:41,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:28:41,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:28:41,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:28:44,181 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:45,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:51,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:51,721 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:28:52,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:56,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:28:56,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:28:56,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:28:56,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:00,288 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:01,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:07,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:11,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:29:11,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:29:11,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:29:12,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:12,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:17,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:21,721 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:29:22,438 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:23,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:26,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:29:26,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:29:26,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:29:29,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:32,514 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:35,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:40,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:41,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:29:41,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:29:41,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:29:42,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:45,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:50,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:51,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:51,721 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:29:56,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:57,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:29:57,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:29:57,157 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:30:00,709 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:02,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:08,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:10,774 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:12,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:30:12,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:30:13,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:18,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:18,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:21,722 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:30:23,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:27,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:30:27,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:30:28,901 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:29,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:34,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:36,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:39,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:42,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:30:42,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:30:42,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:30:44,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:47,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:49,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:51,724 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:30:54,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:30:54,126 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:30:54,127 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:30:54,128 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:30:55,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:30:55,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:55,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:56,115 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:57,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:30:57,113 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:31:00,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:06,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:07,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:11,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:12,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:31:12,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:31:12,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:31:15,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:17,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:21,725 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:31:22,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:25,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:27,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:31:27,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:31:27,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:31:27,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:33,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:34,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:39,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:42,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:31:42,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:31:42,586 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:31:44,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:44,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:49,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:51,725 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:31:54,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:55,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:57,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:31:57,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:31:57,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:32:00,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:04,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:32:06,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:11,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:12,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:32:12,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:32:12,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:32:12,631 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:32:16,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:21,726 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:32:22,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:32:22,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:27,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:28,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:32:28,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:32:28,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:32:30,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log